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EXECUTIVE SUMMARY 


In response to a severe recession that began in 2007, the U.S. Congress passed, and 
President Barack Obama signed into law, the American Recovery and Reinvestment Act of 2009 
(Pub. Law 1 1 1-5). At an estimated cost of $831 billion, this economic stimulus package sought 
to save and create jobs; provide temporary relief to those adversely affected by the recession; and 
invest in education, health, infrastructure, and renewable energy. States and school districts 
received $100 billion to secure teachers’ jobs and promote innovation in schools. This funding 
included $4.35 billion for Race to the Top (RTT), one of the Obama administration’s signature 
programs and one of the largest federal government investments in an education grant program. 
RTT awarded three rounds of grants to states that agreed to implement a range of education 
policies and practices designed to improve student outcomes. In particular, the program sought to 
improve student outcomes for high-need students, including English language learners (ELLs) 
(U.S. Department of Education 2010). 

Given the importance and size of the RTT grant program, the U.S. Department of Education 
(ED) commissioned this evaluation to address the following broad issues: 

• Whether states that received an RTT grant used the policies and practices promoted by RTT 

and how that compares to other states 

• Whether use of these policies and practices included a focus on ELLs and whether that focus 

on ELLs differed between RTT and other states 

• Whether receipt of an RTT grant was related to improvement in student outcomes 

No new funds were appropriated for RTT in the most recent federal budget (for fiscal year 
2016), so the future of the activities and policies that began under RTT is uncertain. However, 
findings on the first and second issues presented in this report remain useful to (a) policymakers 
who are interested in broader lessons learned from the program’s implementation and (b) 
educators who are considering how to proceed with education policy in light of the additional 
flexibility that the Every Student Succeeds Act of 2015 provides to states in terms of school 
turnaround, accountability, assessment, and educator evaluation systems. Findings on the first 
and second issues also provide useful context for interpreting the findings on the third issue. For 
example, if use of the policies and practices promoted by RTT is similar between states that 
received grants and states that did not, then it seems less likely that we would observe a 
relationship between RTT grant receipt and student outcomes. Findings on the third issue remain 
of interest to policymakers and educators who would like to better understand whether and how 
the large RTT investment might be related to changes in student outcomes. 

This is the final report on RTT for this evaluation. An earlier brief focused specifically on 
whether states adopted teacher evaluation policies promoted by RTT in spring 2012, and whether 
adoption of these policies varied across states that did and did not receive RTT grants (Hallgren 
et al. 2014). An earlier report covered all major policy and practice areas that RTT promoted, 
examining the extent to which states reported using these policies and practices in spring 2012, 
and whether usage differed across states that did and did not receive RTT grants (Dragoset et al. 
2015). This final report builds on the earlier brief and report by including an additional year of 
data (spring 2013) and by examining how receipt of RTT grants was related to student 
achievement over time. 
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EXECUTIVE SUMMARY 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Main findings 

We examined the extent to which RTT grantees and other states reported using policies and 
practices in six main areas: (1) improving state capacity to support school improvement efforts; 
(2) adopting standards and assessments that prepare students to succeed in college and the 
workplace; (3) building state data systems that measure student growth and inform instruction; 

(4) recruiting, developing, rewarding, and retaining effective teachers and principals; (5) turning 
around low-performing schools; and (6) encouraging conditions in which charter schools can 
succeed. We conducted two sets of comparisons: (1) we compared Round 1 and 2 RTT states 
(tenned early RTT states in this report) with states that did not receive RTT grants (termed non- 
RTT states or other states in this report) and (2) we compared Round 3 RTT states (tenned later 
RTT states in this report) with non-RTT states. We distinguish between Rounds 1 or 2 and 
Round 3 because of differences in the grants’ timing, funding levels, and scope for these groups 
of states. We found the following: 

• In four of the six areas examined, early RTT states reported using more policies and 
practices promoted by RTT than non-RTT states in spring 2013. The four areas were (1) 
adopting standards and assessments that prepare students to succeed in college and the 
workplace; (2) recruiting, developing, rewarding, and retaining effective teachers and 
principals; (3) turning around low-performing schools; and (4) encouraging conditions in 
which charter schools can succeed. 

• Later RTT grantees reported using more RTT-promoted policies and practices related 
to teacher and principal certification and evaluation than non-RTT states in spring 2013. 

• Across all states, use of RTT-promoted policies and practices was most common for 
data systems and least common for teacher and principal certification and evaluation. 

In the data systems area, states reported using, on average, 76 percent of the RTT-promoted 
practices examined. In the teacher and principal certification and evaluation area, states 
reported using, on average, 26 percent of RTT-promoted practices examined. 

• Across the six areas, early RTT states reported using more English language learner 
(ELL)-focused policies and practices promoted by RTT than non-RTT states. No 

differences were found between later RTT states and non-RTT states. 

• Findings from spring 2012 and spring 2013 on states' use of RTT-promoted policies 
and practices were similar. For the most part, the spring 2013 findings presented in this 
report were the same as the spring 2012 findings presented in an earlier report from this 
evaluation (Dragoset et al. 2015). 

• There were no significant differences between RTT and other states in use of RTT- 
promoted practices over time. When we examined changes over time in states' use of 
RTT-promoted practices, we found no significant differences between RTT and other states. 

• The relationship between RTT and student outcomes was not clear. Trends in student 
outcomes could be interpreted as providing evidence of a positive effect of RTT, a negative 
effect of RTT, or no effect of RTT. 
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In sum, it is not clear whether the RTT grants influenced the policies and practices used by 
states or whether they improved student outcomes. RTT states differed from other states prior to 
receiving the grants, and other changes taking place at the same time as RTT reforms may also 
have affected student outcomes. Therefore, differences between RTT states and other states may 
be due to these other factors and not to RTT. Furthermore, readers should use caution when 
interpreting the results because the findings are based on self-reported use of policies and 
practices. 

Background 

The first three rounds of the RTT grant competition sought to encourage states to implement 
a range of policies and practices designed to affect all levels of the education system, with the 
ultimate goal of improving student outcomes. 1 The six topic areas described in the RTT 
application were (1) improving state capacity to support school improvement efforts; (2) 
adopting standards and assessments that prepare students to succeed in college and the 
workplace; (3) building state data systems that measure student growth and inform instruction; 
(4) recruiting, developing, rewarding, and retaining effective teachers and principals; (5) turning 
around low-perfonning schools; and (6) encouraging conditions in which charter schools can 
succeed. The RTT objectives in each topic area and the subtopics within each topic are detailed 
in Table ES.l. 

The RTT grants were awarded to states that both demonstrated a solid record of reform (for 
example, states that had improved student outcomes overall and by student subgroup and that 
had made progress in the past in the RTT reform areas) and presented strong plans in their RTT 
applications for furthering policies in these areas. Across the first three rounds of competition, 46 
states and the District of Columbia applied for RTT grants, and 19 applicants received grants. 2 
The Round 1 awards were made in March 2010, Round 2 awards in August 2010, and Round 3 
awards in December 2011. The 12 states selected in the first two rounds received awards ranging 
from $75 million to $700 million. 3 In the third round, which was open only to the nine finalists 
from the second round who had not yet received an RTT grant, awards were made to 7 states. 
Because these awards were smaller (ranging from $17 million to $43 million), ED required these 
states to focus on only a portion of the policies described in their Round 2 applications. 


1 Additional rounds of the RTT program focused on improving early learning and development programs for young 
children and supporting district-developed plans to improve student achievement. Those rounds were not a focus of 
this study. 

2 Alaska, North Dakota, Texas, and Vermont did not apply for RTT grants in any round. 

3 States’ award amounts varied based on their share of the nation’s school-age population and the budget they 
proposed in their application for accomplishing their specific plans. 
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Table ES.1. Objectives of the RTT grants, by topic area 


Improving state capacity to support school improvement efforts 


Articulating the state’s education reform agenda and local education agencies’ participation in it 
Building strong statewide capacity to implement, scale up, and sustain the proposed plans 
Demonstrating significant progress in raising achievement and closing gaps 


Adopting standards and assessments that prepare students to succeed in college and the workplace 


Developing and adopting common standards 

Developing and implementing common, high-quality assessments 

Supporting the transition to enhanced standards and high-quality assessments 


Building state data systems that measure student growth and inform instruction 


Fully implementing a statewide longitudinal data system 
Accessing state data and using it to inform key stakeholders 
Using data to improve instruction 


Recruiting, developing, rewarding, and retaining effective teachers and principals 


Providing high-quality pathways to certification for aspiring teachers and principals 
Improving teacher and principal effectiveness based on performance 
Ensuring equitable distribution of effective teachers and principals 
Improving the effectiveness of teacher and principal preparation programs 
Providing effective support to teachers and principals 


Turning around low-performing schools 


Authority to intervene in the lowest-achieving schools and local education agencies 
Turning around the lowest-achieving schools 


Encouraging conditions in which charter schools can succeed 


Eliminating restrictions on charter school creation and enrollment 
Refining authorization and monitoring processes 
Source: RTT application. 
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Research questions and study design 

This report was guided by the following research questions: 

• Are RTT states using the educational policies and practices promoted by RTT, and how does 
that use compare to the use of those policies and practices by other states? 

• Does use of these policies and practices include a focus on ELLs, and does that focus on 
ELLs differ between RTT and other states? Does use of these ELL-focused educational 
policies and practices differ based on characteristics that might affect the relevance of using 
these policies and practices, such as the percentage of ELL students in the state or the 
achievement gap between ELLs and other students? 

• Is receipt of an RTT grant related to improvement in student outcomes? 

The theory of action for RTT is that policy changes at the state level represent the first step 
in the process of changing the education system. Changes must occur at the state level before 
occurring at other levels, such as in districts, schools, and classrooms. For example, for a district 
to change its teacher evaluation system, a state might have to first make changes to its teacher 
evaluation policies and requirements. The changes made at all levels of the education system 
could then improve student achievement. The RTT study design is summarized in the box below. 

Prior to receiving a grant, RTT states differed somewhat from other states 

The RTT program sought to reward states that not only proposed strong reform plans but 
that also had a solid record of reform, so it is possible that RTT states differed from other states 
before they were awarded their grants. We compared these groups before receipt of the grants to 
better isolate changes that may have been due to the grant and found: 

• States that received an RTT grant were already using more of some RTT-promoted 
policies and practices before the grants were awarded. States that received an RTT grant 
received higher scores from grant application reviewers than other states that applied for 
RTT on pre-existing state policy conditions. In addition, RTT states reported using more of 
the policies and practices aligned with the RTT program at baseline than other states in the 
teacher and principal certification and evaluation area. Finally, later RTT states also reported 
using more policies and practices aligned with RTT at baseline in the area of school 
turnaround. 
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RTT study design 

Sample. The sample for the RTT evaluation included 50 states and the District of 
Columbia (DC). 

Data on educational policies and practices. To collect information on states’ use of the 
policies and practices promoted by RTT, we conducted structured telephone interviews with 
representatives from state education agencies. We interviewed 49 states and DC in spring 2012 (for 
a 98 percent response rate) and 50 states and DC in spring 2013 (for a 100 percent response rate). 
There were six interview modules (one for each RTT topic area). Respondents were state 
administrators most knowledgeable about each area. To facilitate comparisons between RTT and 
other states, we asked all states the same questions. During the spring 2012 interviews, we not only 
collected data on the current school year (2011-2012), we also collected data on three of the six 
reform areas in the year before the announcement and implementation of RTT (2007-2008). These 
three areas included (1) teacher and principal certification and evaluation, (2) school turnaround, and 
(3) charter schools. All data provided by states were self-reported and not independently verified by 
the research team. For these reasons and potential concerns about recall accuracy, readers should 
exercise caution when interpreting the data, particularly from 2007-2008. 

Data on student achievement. To examine the relationship between receipt of an RTT grant 
and student outcomes, we obtained publicly available data on state-level test scores from the 
National Assessment of Educational Progress (NAEP), a nationally representative assessment of 
U.S. students. 

Analysis of RTT implementation. To examine how use of policies and practices promoted by 
RTT compares between RTT states and non-RTT states, we conducted two types of comparisons: 

(1) early RTT states (Round 1 and 2) with non-RTT states and (2) later RTT states (Round 3) with 
non-RTT states. We distinguished between early and later RTT states because of differences in the 
grants’ timing, funding levels, and scope between these groups of RTT states. To summarize the 
large amount of data collected, we identified state interview questions that aligned with the policies 
and practices that RTT sought to affect. We determined how many policies and practices each state 
reported using and then calculated the average number of policies and practices for early RTT 
states, later RTT states, and non-RTT states. We then tested whether differences were statistically 
significant between each of the RTT groups and the non-RTT group in the average number of 
policies and practices reported. Because the goal of this analysis was to provide descriptive 
information about the actual levels of policies and practices used by RTT and non-RTT states in 
spring 2012 and spring 2013, the results were reported as raw (unadjusted) means; they were not 
regression-adjusted to account for any pre-existing differences between RTT and non-RTT states. 
Readers should exercise caution when interpreting findings from this analysis because any 
differences in states’ use of policies and practices were not necessarily caused by RTT. Differences 
could be due to other factors, such as pre-existing differences between RTT and non-RTT states or 
other changes that took place at the same time as RTT. 

Analysis of relationship between RTT and student outcomes. Because it was not possible 
to provide credible estimates of the effect of RTT on student outcomes, we conducted a descriptive 
analysis of student outcomes before and after the award of RTT grants. This analysis plotted the 
average outcomes for early RTT states, later RTT states, and non-RTT states in each year. As with 
the analysis of RTT implementation, readers should exercise caution when interpreting these findings 
because any differences in student outcomes were not necessarily caused by RTT. 
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In spring 2013, early RTT states reported using more policies and practices 
promoted by RTT than non-RTT states in four of the six areas examined 

In spring 2013, early RTT states, on average, reported using more policies and practices than 
non-RTT states in four areas (Figure ES.l): 4 

• Standards and assessments (72 percent of policies and practices reported by early RTT states 
compared to 46 percent for non-RTT states) 

• Teacher and principal certification and evaluation (37 percent of policies and practices 
reported by early RTT states compared to 20 percent for non-RTT states) 

• School turnaround (65 percent of policies and practices reported by early RTT states 
compared to 53 percent for non-RTT states) 

• Charter schools (6 1 percent of policies and practices reported by early RTT states compared 
to 4 1 percent for non-RTT states) 

The magnitude of these differences was between one and three practices for all but one area 
(teacher and principal certification and evaluation), where the difference was larger (about seven 
practices). Therefore, some of the differences may not be meaningful in tenns of how they 
affected education policy and outcomes in the states. 

The differences between RTT and other states observed in 2013 may not be due to the RTT 
grant but rather to differences in state education policy prior to grant award. In the teacher and 
principal certification and evaluation area, for example, early RTT states reported using more of 
the policies and practices promoted by RTT than non-RTT states in spring 2013, but also in the 
2007-2008 school year, before the RTT grants were awarded. This pre-existing difference 
suggests that RTT grants were awarded to states that were already using some of the promoted 
policies and practices rather than RTT awards causing the use of those policies and practices. 5 

In one of the six areas examined, teacher and principal certification and 
evaluation, later RTT states reported using more policies and practices 
promoted by RTT than non-RTT states in spring 2013 

In spring 2013, later RTT states reported using more policies and practices than non-RTT 
states in one of six areas promoted by RTT (the area of teacher and principal certification and 
evaluation) (Figure ES. I). 6 Later RTT states reported using 33 percent of policies and practices 
in this area, on average, compared to 20 percent for non-RTT states. The magnitude of this 


4 Chapter IV contains results for the individual policies and practices within each of these areas. 

5 For three areas (state capacity, standards and assessments, data systems), we do not have information on use of 
policies and practices prior to the awarding of RTT grants. Therefore, we cannot determine whether the observed 
differences between early RTT states and non-RTT states in these areas predated receipt of the RTT grant. 

6 Because Round 3 grants were narrower in scope, not all later RTT states focused on every area targeted by RTT. In 
addition, less time had elapsed between receipt of RTT awards and our spring 2013 interviews for the later RTT 
states than for the early RTT states. Because of these reasons and the smaller sample size (7 later RTT states, as 
opposed to 12 early RTT states), statistically significant differences are less likely to be found between later RTT 
states and non-RTT states than between early RTT states and non-RTT states. 
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Figure ES.1. Use of policies and practices promoted by RTT, by topic area 



(10 items) Assessments (8 items) and Principal Turnaround Schools 

(10 items) Certification (10 items) (4 items) 

and Evaluation 
(39 items) 


Early RTT States (N=12) 


Later RTT States (N=7) 


Non-RTT States (N=32) 


Source: Interviews with state administrators in spring 2013. 

Note: The total number of policies and practices (shown in parentheses below each bar) differs by topic area. 

The number of policies and practices that we examined for each area was directly related to the amount 
of emphasis placed on each area in the RTT application criteria. This figure reads as follows (using the 
first bar on the left as an example): early RTT states reported using 71 percent of the policies and 
practices in the state capacity area. 

* Significantly different from non-RTT states at the 0.05 level, two-tailed test. 


difference was slightly more than five practices. As with the early RTT states, baseline data 
suggest a difference between these groups in this area existed before RTT grants were awarded. 


Across all states, use of policies and practices promoted by RTT was highest 
for data systems and lowest for teacher and principal certification and 
evaluation 

Use of RTT-promoted policies and practices was highest in the data systems area, in which 
states reported using, on average, 76 percent of the RTT-promoted practices examined (not 
shown). Use of RTT-promoted policies and practices was lowest in the teacher and principal 
certification and evaluation area. In that area, states reported using, on average, 26 percent of the 
RTT-promoted practices examined (not shown). When focusing on use of individual policies and 
practices, nearly ah states reported (1) having a state longitudinal data system (SLDS), and (2) 
identifying teacher shortage areas (not shown). In contrast, no states reported using the following 
policies and practices: (1) using results from evaluations of certification programs to provide 
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additional funds for, expand, or promote certification programs that were shown to be effective 
for teachers; (2) doing the same for certification programs that were shown to be effective for 
principals; and (3) publicly reporting results from evaluations of certification program 
effectiveness for principals. 

Across the six areas, early RTT states reported using more ELL-focused 
policies and practices promoted by RTT than non-RTT states in spring 2013, 
but there were no differences between later RTT and non-RTT states 

Early RTT states reported using more ELL-focused policies and practices promoted by RTT 
than non-RTT states. Out of the 12 ELL-focused policies and practices, early RTT states 
reported using 58 percent (7.0 practices) compared to 45 percent (5.4 practices) for non-RTT 
states. Later RTT states reported using 53 percent of these policies and practices (6.4 practices), 
which did not significantly differ from non-RTT states. 

Findings from spring 2012 and spring 2013 on states’ use of RTT-promoted 
policies and practices were similar 

Lor the most part, the spring 2013 findings presented in this report were the same as the 
spring 2012 findings presented in an earlier report from this evaluation (Dragoset et al. 2015). 
Two key differences were (1) early RTT states reported using more policies and practices than 
non-RTT states in five out of six areas in spring 2012 (with school turnaround being the 
exception), compared to four out of the six areas in spring 2013 (with state capacity and data 
systems being the exceptions); and (2) in spring 2012, there were no differences between RTT 
states and other states in use of ELL-focused policies and practices, but in spring 2013, early 
RTT states reported using more of these policies and practices than non-RTT states. 

There were no significant differences between RTT and other states in use of 
RTT-promoted policies and practices over time 

When we examined changes over time in states’ use of RTT-promoted policies and 
practices, we found no significant differences between RTT and other states. This finding was 
the same for each of the six topic areas examined and for the ELL-focused policies and practices. 

The relationship between RTT and student outcomes was not clear 

The trends in outcomes that we observed could be interpreted as providing evidence of a 
positive effect of RTT, a negative effect of RTT, or no effect of RTT. Therefore, it is not clear 
whether receipt of an RTT grant was related to changes in student outcomes. This uncertainty is 
partly due to the difficulty in separating the effect of RTT from overall trends in student 
outcomes and partly due to the limited amount of data available for the analysis. 

Conclusions 

It is not clear whether the RTT program influenced the use of policies and practices 
promoted by the program in RTT states. Although some differences between RTT and other 
states were observed, other factors could explain those differences. In particular, some 
differences in use of policies and practices promoted by RTT existed prior to states’ receipt of 
RTT grants. 
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Similarly, it is not clear whether RTT influenced student outcomes. This uncertainty exists 
because (a) other changes taking place at the same time as RTT reforms might also have affected 
student outcomes, and (b) the findings could be interpreted as providing evidence of RTT having 
a positive effect, negative effect, or no effect. 



I. INTRODUCTION 


The American Recovery and Reinvestment Act of 2009 (ARRA) provided an unprecedented 
amount of federal funding for education in an effort to mitigate the effects of the nation’s 
economic downturn and to make a lasting investment in education. Through $97.4 billion in 
ARRA funds, the federal government sought to save education jobs, fund a new wave of 
innovation in education, and support comprehensive efforts to turn around low-perfonning 
schools. Race to the Top (RTT), a new competitive state grant program authorized under ARRA, 
received $4.35 billion to spur states to use a broad array of policies and practices designed to 
affect all levels of public school education with the ultimate goal of improving student outcomes. 
The RTT application criteria described six primary topic areas: (1) improving state capacity to 
support school improvement efforts; (2) adopting standards and assessments that prepare 
students to succeed in college and the workplace; (3) building state data systems that measure 
student growth and success and inform instruction; (4) recruiting, developing, rewarding, and 
retaining effective teachers and principals; (5) turning around low-perfonning schools; and (6) 
encouraging conditions in which charter schools can succeed. 

The U.S. Department of Education’s (ED’s) Institute of Education Sciences (IES) 
commissioned an evaluation of the RTT program. The RTT evaluation is based on a descriptive 
analysis of state-level education policies and practices in the six areas listed above, and a 
descriptive analysis that compares student outcomes in states that received RTT grants with 
student outcomes in states that did not. 

At the request of ED’s Office of English Language Acquisition (OELA), part of the 
evaluation also focuses on how states have addressed the needs of English language learners 
(ELLs) as they used the policies and practices promoted by RTT. ELLs are of particular interest 
to this evaluation because: (1) they are historically lower-achieving than non-ELLs, 7 and (2) the 
RTT program sought to prioritize the academic achievement of high-needs students, including 
ELLs (U.S. Department of Education 2010). 8 This report examines states’ use of ELL-focused 
policies and practices across the six general areas listed above. The report examines use of ELL- 
focused policies and practices overall, as well as among subgroups of states for which ELL- 
focused education reforms might have been a particularly high priority: states with higher ELL 
populations and those with higher achievement gaps between ELLs and other students. 
Examining these subgroups could illuminate important differences between RTT and other states 
that only exist among states with a greater need for ELL-focused education reforms. 

This is the final report on RTT for this evaluation. An earlier brief focused specifically on 
whether states adopted teacher evaluation policies promoted by RTT in spring 2012, and whether 
adoption of these policies varied across states that did and did not receive RTT grants (Hallgren 
et al. 2014). An earlier report covered all six primary policy and practice areas that RTT 
promoted, examining the extent to which states reported using these policies and practices in 


7 Since 2002, ELLs’ reading test scores have been below those of non-ELLs on the National Assessment of 
Educational Progress test (National Center for Education Statistics 2014). 

8 In particular, the RTT application criteria asked states to (1) make data on ELLs available to evaluate the 
effectiveness of instructional materials, strategies, and approaches for ELLs, and (2) provide supports and 
professional development to teachers and principals to ensure that ELLs acquire language skills to master academic 
content (U.S. Department of Education 2010). 
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spring 2012, and whether usage differed across states that did and did not receive RTT grants 
(Dragoset et al. 2015). This final report builds on the earlier brief and report by including an 
additional year of data (spring 2013) and by examining how receipt of RTT grants was related to 
student achievement over time. 

In this chapter we provide background information about the RTT program, summarize prior 
research on RTT, and provide an overview of our evaluation and the contents of this report. 

A. Scope, purpose, timing, and size of RTT grants 

RTT was designed to promote systemic change in the U.S. educational system, with the 
ultimate goal of improving student achievement. ED awarded three types of RTT grants between 
2010 and 2013: Round 1, 2, and 3 grants funded by ARRA; RTT-Early Learning Challenge 
grants; and RTT-District grants. 9 This evaluation focuses on the Round 1, 2, and 3 grants to 
states that were funded by ARRA. In the remainder of this report, when we refer to RTT, we are 
referencing the Round 1 to 3 RTT grants and not the other two types. 

RTT aims to facilitate policies and practices in the six areas listed above. RTT grants were 
designed to encourage and reward states that had strong records of using these policies and 
practices and that presented plans in their RTT applications for furthering policies and practices 
in these areas (U.S. Department of Education 2010). 

The Race to the Top Executive Summary (U.S. Department of Education 2009) summarizes 
the program’s priorities and theory of action. The general theory of action underlying RTT is that 
policy changes at the state level represent the first step toward changing the education system 
with the ultimate goal of improving student outcomes. Changes would be expected to occur at 
the state level before occurring at lower levels, such as districts, schools, and classrooms. The 
changes made at all levels of the education system could then improve student achievement. 

RTT grants were awarded through a competition open to all states, the District of Columbia, 
and Puerto Rico. To maximize the program’s likelihood of fostering systemic reforms, ED 
required states applying for RTT grants to demonstrate substantial commitment to implement 
their proposed plans at the local level. In their RTT applications, states had to identify the local 
education agencies (LEAs) committing to implement all or the vast majority of the policies and 
practices proposed by the state — that is, participating LEAs. In contrast, involved LEAs were 
those committing to participate in specific portions of RTT policies and practices that would 
necessitate statewide or nearly statewide implementation (for example, adopting common 
standards). ED required RTT states to distribute no less than half of their grant funds to 


9 RTT-Early Learning Challenge grants totaling $913 million were made to 20 states in 2011, 2012, and 2013, with 
the goal of improving early learning and development programs so that children entering kindergarten would be 
ready to succeed. The program focuses on improving early learning and development for young children in two 
ways: (1) by supporting states’ efforts to increase the number and percentage of disadvantaged children enrolled in 
high-quality early learning and development programs and (2) by designing and implementing an integrated system 
of high-quality early learning and development programs and services. RTT-District grants totaling $520 million 
were made to 21 school districts in December 2012 and 2013 to support locally developed plans with four goals: (1) 
to personalize and deepen student learning, (2) to directly improve student achievement and educator effectiveness, 
(3) to close achievement gaps, and (4) to prepare every student to succeed in college and their careers. 
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participating LEAs, apportioned to them according to the LEAs’ share of Title I funds. 10 RTT 
states were allowed substantial flexibility regarding the use of the balance of their grants. They 
could use the remaining funds to support statewide activities (such as enhancements to data, 
evaluation, or support systems), as supplementary awards to participating LEAs, as sub-grants to 
involved LEAs, or for other activities. 

Across the three rounds of competition, 46 states and the District of Columbia applied for 
RTT grants. (The exceptions were Alaska, North Dakota, Texas, and Vermont.) The timing of 
awards, number of states selected, range of award amounts, and other descriptive information 
about each round are shown in Table 1.1. States’ award amounts varied based on their share of 
the nation’s school-age population and the budget they proposed in their application for 
accomplishing the specific plans they proposed. In this report, we refer to the 12 states selected 
in the first two rounds as early RTT states. We refer to the 7 states selected in the third round as 
later RTT states. The early and later states were judged by ED to have articulated comprehensive 
plans to improve education policies and practices and to have demonstrated the capacity for, and 
significant past progress in, implementing policies and practices in the program’s topic areas. 

The RTT applicants, their application scores from Rounds 1 and 2, the award amounts, and the 
percentage of LEAs participating for each grantee are listed in Table 1.2. 11 States have four years 
from the time of the award to implement their plans and spend the grant funds, although states 
that anticipate needing additional time to complete certain projects were permitted to request a 
no-cost extension (as of September 2015, Delaware, the District of Columbia, Florida, Georgia, 
Maryland, Massachusetts, New York, North Carolina, Ohio, Pennsylvania, Rhode Island, and 
Tennessee had received approval for no-cost extensions). Note that when we refer to RTT states, 
we are including the District of Columbia. 


Table 1.1. Information about RTT competition rounds 1, 2, and 3 



Round 1 

Round 2 

Round 3 

Eligible applicants 

Open to all states, the 
District of Columbia, and 
Puerto Rico 

Open to all states other 
than Round 1 grantees 
and to the District of 
Columbia and Puerto Rico 

Open to only the nine 
finalists from Round 2 that 
did not receive a grant in 
Round 2 

Policies and practices 

ED asked states to 
address policies and 
practices in all six areas 
listed in the RTT 
application criteria 

ED asked states to 
address policies and 
practices in all six areas 
listed in the RTT 
application criteria 

ED required states to 
focus on only a portion of 
the policies and practices 
in their Round 2 
applications 3 

Budget 

ED provided suggested 
budget ranges for each 
state 

States required to stay 
within the Round 1 budget 
ranges provided 

ED provided suggested 
budget amounts for each 
state 

Timing of awards 

March 2010 

August 2010 

December 201 1 

Number of states selected 

2 

10 

7 


10 Title I, Part A (Title I) of the Elementary and Secondary Education Act provides financial assistance to LEAs and 
schools with many children from low-income families. Title 1 funds are allocated using formulas that are based 
primarily on census poverty estimates and the cost of education in each state. 

11 Application scores provide important context about ways in which RTT states and non-RTT states may have 
differed before RTT awards were announced. In Chapter III, we present additional analysis of the average scores for 
each group of states. ED did not calculate scores in Round 3. 
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Round 1 

Round 2 

Round 3 | 

Award amounts 

$100 million-$500 million 

$75 million— $700 million 

$17 million— $43 million 


Source: U.S. Department of Education. 

Note: Alaska, North Dakota, Texas, and Vermont did not apply for RTT grants in any round. 

ED = U.S. Department of Education. 


a All seven Round 3 states focused on policies and practices in the areas of state capacity and standards and 
assessments. Six states also focused on policies and practices in the areas of data systems (Colorado was the 
exception) and teachers and principals (Arizona was the exception). One state (Louisiana) also focused on policies 
and practices in the area of school turnaround. Two states (New Jersey and Pennsylvania) also focused on policies 
and practices in the area of charter schools. 


Table 1.2. Information about RTT applicants and grantees 


State 

Round of 
award 

Round 1 

score 

Round 2 

score 

Award 

amount 

Percentage of 
LEAs 

participating 

Delaware 

1 

455 

a 

$100 million 

100 

Tennessee 

1 

444 

a 

$500 million 

100 

Georgia 

2 

434 

446 

$400 million 

14 

Florida 

2 

431 

452 

$700 million 

96 

Rhode Island 

2 

419 

451 

$75 million 

96 

Ohio 

2 

419 

441 

$400 million 

53 

North Carolina 

2 

414 

442 

$400 million 

100 

Massachusetts 

2 

411 

471 

$250 million 

70 

New York 

2 

409 

465 

$700 million 

86 

District of Columbia 

2 

402 

450 

$75 million 

63 

Hawaii 

2 

365 

462 

$75 million 

100 

Maryland 

2 

a 

450 

$250 million 

92 

Illinois 

3 

424 

427 

$43 million 

C 

Pennsylvania 

3 

420 

418 

$41 million 

c 

Kentucky 

3 

419 

412 

$17 million 

c 

Louisiana 

3 

418 

434 

$17 million 

72 

Colorado 

3 

410 

420 

$18 million 

51 

New Jersey 

3 

387 

438 

$38 million 

28 

Arizona 

3 

240 

435 

$25 million 

C 

South Carolina 

b 

423 

431 

b 

b 

Arkansas 

b 

394 

389 

b 

b 

Utah 

b 

379 

379 

b 

b 

Minnesota 

b 

375 

a 

b 

b 

Michigan 

b 

366 

382 

b 

b 

Indiana 

b 

356 

a 

b 

b 

Iowa 

b 

346 

383 

b 

b 

Connecticut 

b 

345 

379 

b 

b 

Wisconsin 

b 

341 

368 

b 

b 

California 

b 

337 

424 

b 

b 

Idaho 

b 

331 

a 

b 

b 

Kansas 

b 

330 

a 

b 

b 

New Mexico 

b 

325 

366 

b 

b 

Virginia 

b 

325 

a 

b 

b 

Wyoming 

b 

319 

a 

b 

b 

Missouri 

b 

301 

316 

b 

b 

Oklahoma 

b 

295 

392 

b 

b 

Oregon 

b 

293 

a 

b 

b 

West Virginia 

b 

292 

a 

b 

b 

Alabama 

b 

291 

212 

b 

b 

New Hampshire 

b 

271 

335 

b 

b 

Nebraska 

b 

247 

296 

b 

b 

South Dakota 

b 

136 

a 

b 

b 

Maine 

b 

a 

283 

b 

b 
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State 

Round of 
award 

Round 1 

score 

Round 2 

score 

Award 

amount 

Percentage of 
LEAs 

participating 

Mississippi 

b 

a 

263 

b 

b 

Montana 

b 

a 

238 

b 

b 

Nevada 

b 

a 

381 

b 

b 

Washington 

b 

a 

291 

b 

b 


Source: U.S. Department of Education. 


Note: States are listed within rounds by their Round 1 score. Alaska, North Dakota, Texas, and Vermont did not 

apply for RTT grants in any round. The U.S. Department of Education did not calculate scores in Round 3. 
a Delaware and Tennessee do not have Round 2 scores because they received grants in Round 1. Maryland, Maine, 
Mississippi, Montana, Nevada, and Washington do not have Round 1 scores because they did not apply for an RTT 
grant in Round 1. Minnesota, Indiana, Idaho, Kansas, Virginia, Wyoming, Oregon, West Virginia, and South Dakota 
do not have a Round 2 score because they did not apply for an RTT grant in Round 2. 

b These states did not receive a grant in any round. 

c These states did not report the percentage of LEAs participating or provide a list of participating LEAs in their RTT 
Round 3 applications. 

LEA = local education agency. 

B. Prior research on RTT reported both progress and challenges 

Given that RTT is a relatively new program, to our knowledge no studies have examined the 
impact of RTT on student outcomes or other measures of educational improvement. Several 
studies, however, have examined RTT implementation progress in RTT states. Three main 
themes emerged from these studies: 

1. RTT states made progress in implementing RTT policies and practices (Boser 2012; Fain 
and Haxton 2014; Government Accountability Office 2011; Government Accountability 
Office 2015; Miller and Hanna 2014; Troppe et al. 2015; U.S. Department of Education 
2015; U.S. Department of Education 2016b). 12 

2. RTT states encountered challenges in their implementation of RTT policies and practices, 
including some delays and difficulties implementing certain types of policies and practices. 
In particular, many states experienced difficulties implementing teacher and principal 
evaluation systems and developing data systems (Boser 2012; Government Accountability 
Office 2011; Government Accountability Office 2013; Government Accountability Office 
2015; Miller and Hanna 2014; Russell et al. 2015; Troppe et al. 2015; U.S. Department of 
Education 2016b; Weiss 2013). 

3. RTT states were more likely than other states to use RTT-promoted policies and practices 
(Dragoset et al. 2015; Government Accountability Office 2011; Howell 2015; National 
Council on Teacher Quality 2011). There was also some evidence that policy adoption rates 
increased more for RTT states than other states in the years after RTT grants were awarded 
(Howell 2015). 

C. Prior research on the relationship between RTT-promoted policies and 
student outcomes is not conclusive 

A vast literature examining the effectiveness of the types of policies promoted by RTT 
provides no conclusive evidence on whether they improve student outcomes. In two of the six 


12 U.S. Department of Education (2016b) provides RTT annual performance reports for each state. 
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areas described in the RTT application criteria (school turnaround and standards and 
assessments), to our knowledge, no experimental studies have examined the relationship between 
RTT-promoted policies and practices and student outcomes, and non-experimental studies found 
mixed results. (Some studies found that RTT-promoted policies and practices were associated 
with improved student outcomes, while other studies found no relationship between these 
policies and practices and student outcomes .) 13,14 In three areas (data systems, teacher and 
principal certification and evaluation, and charter schools), the evidence from both experimental 
and non-experimental studies was mixed . 15 In the area of state capacity, to our knowledge, no 
studies examined the relationship between the types of policies and practices promoted by RTT 
and student outcomes. 

D. Evaluation focus 

As noted above, there is no conclusive evidence on whether the RTT program (or the 
policies and practices promoted by the program) improves student outcomes. In addition, few 
studies on the implementation of RTT-promoted policies and practices comprehensively examine 
all the areas described in the RTT application, and few examine whether the policies and 
practices used by RTT states differ from those used by other states. This evaluation seeks to 
address these gaps in the literature. It is designed to examine the relationship between receipt of 
an RTT grant and student achievement, whether RTT grant recipients report using the policies 
and practices promoted by the program, and whether they are doing so to a different extent than 
states that have not received such grants. This report focuses on the following research questions: 

• Are RTT states using the educational policies and practices promoted by RTT, and how does 
that compare to the use of those policies and practices by other states? 

• Does use of these policies and practices include a focus on ELLs, and does that focus on 
ELLs differ between RTT and other states? Does use of these ELL-focused educational 
policies and practices differ based on characteristics that might affect the relevance of using 
these policies and practices, such as the percentage of ELL students in the state or the 
achievement gap between ELLs and other students? 

• Is receipt of an RTT grant related to improvement in student outcomes? 


13 Experimental studies are those that examine treatment and comparison groups from a randomized controlled trial. 
The results from experimental studies (if such studies are conducted appropriately) can be used to make causal 
statements about the effect of a practice, policy, program, or intervention on an outcome of interest. The results from 
non-experimental studies cannot be used to make such causal statements because these studies cannot rule out the 
possibility that other differences between the groups — besides the intervention itself — caused any observed 
differences in outcomes. 

14 To cite a few examples of these non-experimental studies: Ballou and Springer 2008; Bandeira de Mello 2011; 
Bifulco et al. 2003; Booker et al. 2009; Borman et al. 2003; Dee 2012; Dee and Jacob 2011; de la Torre and Gwynne 
2009; Figlio et al. 2009; Fryer 2014; Fuller et al. 2007; Flanushek and Raymond 2005; Lee 2006; Player and Katz 
2013; and Strunk et al. 2012. 

15 To cite a few examples: Abdulkadiroglu et al. 2011; Allen et al. 2011; Carlson et al. 2011; Clark et al. 2013; 
Constantine et al. 2009; Decker et al. 2004; Dobbie and Fryer 2011; Fryer et al. 2012; Garet et al. 2008; Garet et al. 
2010; Glazerman et al. 2006; Glazerman et al. 2013; Gleason et al. 2010; Flenderson et al. 2007; Flenderson et al. 
2008; Floxby et al. 2009; Floxby and Rockoff2005; May and Robinson 2007; Quint et al. 2008; Slavin et al. 2011; 
Tuttle et al. 2013; Zimmer and Buddin 2006; and Zimmer et al. 2012. 
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To address the first two research questions in this report, we conducted interviews with 
state-level administrative staff in 50 states and the District of Columbia in spring 2013. To 
answer the third research question, we examined data on student outcomes for the time periods 
before and after the award of RTT grants. A prior report from this evaluation (Dragoset et al. 
2015) examined the first two research questions based on data from spring 2012. 

E. Report structure 

In Chapter II, we describe the study sample, design, and data collected to address these 
research questions. In Chapter III, we provide baseline infonnation on the RTT sample. In 
Chapter IV, we present findings on the extent to which RTT states reported using the policies 
and practices promoted by RTT in spring 2013 and how that compares to the use of those 
policies and practices by other states. In Chapter V, we present findings on the extent to which 
RTT and other states reported focusing on ELLs in their use of the policies and practices 
promoted by RTT, and how that varies by the percentage of students who are ELLs and the 
achievement gap between ELLs and other students. In Chapter VI, we discuss the findings from 
this report. In Appendices A through E, we provide additional results, including responses to 
individual interview questions (Appendix B) and a descriptive analysis of the relationship 
between receipt of an RTT grant and student achievement (Appendix E). 
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In this chapter, we describe the study sample, the data collected, and the method of 

analyzing the data for the evaluation of RTT. 

A. Study sample and data collection 

The sample for the evaluation of RTT includes 50 states and the District of Columbia (DC). 

The evaluation used several sources of data: 

• State interviews. To collect information about educational policies, practices, and supports 
in the six RTT topic areas, we conducted structured telephone interviews with 
representatives from state education agencies. We interviewed 49 states and DC in spring 
2012 (for a 98 percent response rate; Texas did not participate in the interviews) and 50 
states and DC in spring 2013 (for a 100 percent response rate). The spring 2012 and 2013 
interviews took place roughly 25 and 37 months after Round 1 grants were received, 20 and 
32 months after Round 2 grants were received, and 4 and 16 months after Round 3 grants 
were received. The RTT objectives in each area and the policies and practices within each 
area for which we had state interview data are detailed in Table II. 1 . 16,17,18 The state 
interview included questions addressing 12 ELL-focused policies and practices aligned with 
RTT objectives (Table II. 2). There were six interview modules (one for each RTT topic 
area). The emphasis (that is, the number of interview questions) for each area directly 
related to how much emphasis was provided in the RTT application criteria for each area. 
We typically interviewed different respondents from each state for each module. 
Respondents were state administrators most knowledgeable about each area. To facilitate 
comparisons between RTT and other states, we asked all states the same questions. 


16 Mathematica Policy Research (2012) and Mathematica Policy Research (2014) contain the state interview 
protocols for spring 2012 and 2013. 

17 .. ...... 

The interview questions specified particular time periods of interest. For example, some questions asked, 
“Currently, are you...?” and others asked “In the 2012-2013 school year, did you...?” Therefore, the results in this 
report represent a point-in-time measure of the policies and practices used by states. Throughout the report, we use 
phrases such as “In spring 2013, states reported using policies and practices” to reflect that these are point-in-time 
measures. In particular, the results from spring 2013 represent only the policies and practices used between spring 

2012 and spring 2013, not the total set of policies and practices used at any point after receipt of an RTT grant. 

Some policies and practices (such as creating a new office or department to support the implementation of state 
education reform priorities) might be one-time events. In such cases, a state that used the policy or practice in the 
2011-2012 school year and did not use it again in the following school year (for example, because the new 
office/department was already created in 2011-2012) might have responded “yes” to the question on the spring 2012 
interview and “no” on the spring 2013 interview. Therefore, any observed decreases between 2011-2012 and 2012- 

2013 in the number of policies and practices used may be because certain policies and practices that were one-time 
events were used in 2011-2012 and did not need to be used again the following year. Finally, it is important to keep 
in mind that because the use of policies and practices is a dynamic process, some policies and practices used by 
states at the time of the interview may no longer be in use after the interview. For example, some states that had 
adopted Common Core standards at the time of the interview may have repealed them later. 

15.. .. 

To limit the interview length, we sought input from IES and the RTT program office on which questions were of 
greatest interest. 
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Table 11.1. RTT objectives and policies and practices addressed by state 
interview questions, by topic area 





Policies and practices addressed by state 

Topic area 

RTT objective 


interview questions 

Improving state capacity 

Articulating the state’s education 

• 

Have reform plan in place 

to support school 

reform agenda and local education 

• 

Prioritize college and career-ready standards or 

improvement efforts 

agencies’ participation in it a 


assessments 



• 

Prioritize development of state longitudinal data system 
(SLDS) or data use 



• 

Prioritize teacher and principal evaluation and 
effectiveness 



• 

Prioritize school turnaround 


Building strong statewide capacity 

• 

Support implementation of reform plan 


to implement, scale up, and 

• 

Work with districts to replicate best practices statewide 


sustain the proposed plans 

• 

Provide effective and efficient grant oversight 



• 

Involve teachers, unions, or administrators to define 
priorities 



• 

Involve other stakeholders to define priorities 


Demonstrating significant progress 

No items in state interview aligned with this objective 11 


in raising achievement and closing 
gaps 




Adopting standards and 
assessments that prepare 
students to succeed in 
college and the workplace 


Developing and adopting common 
standards 

Developing and implementing 
common, high-quality 
assessments 


• Adopt Common Core State Standards (CCSS) 


Participate in CCSS Consortium 

Fully implement assessments developed by consortium 


Supporting the transition to 
enhanced standards and high- 
quality assessments 


Provide funds for new staff or new technology to implement 
CCSS 

Provide new technology for CCSS assessments 
Require new curricula or provide materials to implement 
CCSS 


Require or support development or implementation of 
interim assessments 

Change high school exit exam, credit, or college entrance 
requirements 

Provide support or professional development to teachers or 
principals to implement CCSS 
Support CCSS implementation for ELLs 


Building state data Fully implementing an SLDS 

systems that measure 
student growth and inform 
instruction 


Have SLDS 

SLDS has program participation data 
SLDS linked to early childhood data system 
SLDS linked to higher education data system 


Accessing state data and using it 
to inform key stakeholders 


Allow SLDS to be accessed by stakeholders 
Use SLDS to support decision makers 


Recruiting, developing, 
rewarding, and retaining 
effective teachers and 
principals 


Using data to improve instruction 


Providing high-quality pathways to 
certification for aspiring teachers 
and principals 


• Require districts to implement data system 

« Provide supports to districts for using data 

• Authorize qualified alternative-route program operators 0 

• Increase alternative-route program selectivity 

• Increase time in school-based learning experiences 

• Allow alternative-route programs to award same 
certification as traditional-route programs' 1 

• Have alternative-route programs currently operating 

• Have process to identify teacher and principal shortage 
areas 

• Take steps to address teacher and principal shortage 
areas 
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Policies and practices addressed by state 

Topic area 

RTT objective 

interview questions 


Improving teacher and principal 
effectiveness based on 
performance 


Ensuring equitable distribution of 
effective teachers and principals 


Improving the effectiveness of 
teacher and principal preparation 
programs 

Providing effective support to 
teachers and principals 

Turning around low- Having authority to intervene in the 

performing schools lowest-achieving schools and local 

education agencies 

Turning around the lowest- 
achieving schools 


Encouraging conditions in Eliminating restrictions on charter 
which charter schools can school creation and enrollment 
succeed 

Refining authorization and 
monitoring processes 


• Require student achievement growth 

• Require multiple evaluation measures 

• Specify required minimum number of rating levels 

• Conduct annual teacher evaluations® 

• Require that evaluation results be used for professional 
development 

• Require that evaluation results be used for compensation 
decisions 

• Require that evaluation results be used for career 
advancement decisions 

• Require that evaluation results be used for dismissal 
decisions 

• Require use of strategies for more equitable distribution of 
effective teachers and principals 

• Conduct analyses of shifts in distribution of effective 
teachers and principals 

• Use effectiveness ratings to assess certification programs 

• Publicly report certification program effectiveness 

• Support effective certification programs 

No items in state interview aligned with this objective 11 


• Have authority to take over failing schools 


• Provide training in analyzing data to improve instruction 

• Help align curricula to state standards 

• Provide training on school improvement plans, effective 
curricula, instructional strategies, or intervention models 

• Provide training on strategies for ELLs 

• Provide technical assistance on improving professional 
development 

• Provide operational flexibility for staffing and budgeting 

• Provide technical assistance on recruiting and retaining 
teachers or financial incentives 

• Have teacher tenure rules that address placement in or 
removal from lowest-achieving schools 

• Have administrative structures for supporting turnaround 
efforts, monitoring, or providing support 

• Have no restrictions on new charter schools or charter 
enrollment 

• Prioritize schools that address needs of ELLs 

• Monitor academic performance of charter schools 

• Monitor nonacademic performance of charter schools 


Source: RTT application; interviews with state administrators in spring 2012 and spring 2013. 

ELL = English language learner. 

a Regular school districts are the most common type of local education agency. 

b The number of questions included in the state interview was purposefully limited to reduce the time it took to 
complete the interview. We initially developed the interview questions based on an examination of the RTT 
application criteria. To ensure that the interview was of a reasonable length, we then pared down the initial list of 
questions through a deliberative process with the Institute of Education Sciences and the RTT Program Office, to 
assess their priorities for the types of questions to include. The interview did not include any questions about this 
objective. 

c Alternative-route programs are those that offer an alternative route to certification. 
d Traditional-route programs are those that offer a traditional route to certification. 
e The state interview did not ask about this policy or practice for principals. 
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Table 11.2. RTT objectives and the ELL-focused policies and practices aligned 
with those objectives that were addressed by state interview questions, by 

topic area 



Topic area 

RTT objective 

ELL-focused policies and practices addressed by 
state interview questions 

Improving state capacity 
to support school 
improvement efforts 

Articulating the state’s education 
reform agenda and local 
education agencies’ participation 
in it a 

• Prioritize support to ELLs 


Building strong statewide 
capacity to implement, scale up, 
and sustain the proposed plans 

• Provide targeted support to ELLs 

• Implement organizational or administrative changes to improve 
capacity to support ELLs 


Demonstrating significant 
progress in raising achievement 
and closing gaps 

No items in state interview aligned with this objective for ELLs b 

Adopting standards and 
assessments that prepare 
students to succeed in 
college and the workplace 

Developing and adopting 
common standards 

Developing and implementing 
common, high-quality 
assessments 

No items in state interview aligned with this objective for ELLs b 

No items in state interview aligned with this objective for ELLs b 


Supporting the transition to 
enhanced standards and high- 
quality assessments 

• Support Common Core State Standards implementation for 

ELLs 

Building state data 
systems that measure 
student growth and inform 
instruction 

Fully implementing a statewide 
longitudinal data system 

Accessing state data and using it 
to inform key stakeholders 

• State longitudinal data system has program participation data 
about ELLs 

• Use state longitudinal data system to support decision makers 
in improvement efforts for ELLs 


Using data to improve instruction 

• Provide supports to districts for using ELL-related data 

Recruiting, developing, 
rewarding, and retaining 
effective teachers and 
principals 

Providing high-quality pathways 
to certification for aspiring 
teachers and principals 

Improving teacher and principal 
effectiveness based on 
performance 

No items in state interview aligned with this objective for ELLs b 

No items in state interview aligned with this objective for ELLs b 


Ensuring equitable distribution of 
effective teachers and principals 

No items in state interview aligned with this objective for ELLs b 


Improving the effectiveness of 
teacher and principal preparation 
programs 

No items in state interview aligned with this objective for ELLs b 


Providing effective support to 
teachers and principals 

No items in state interview aligned with this objective for ELLs b 

Turning around low- 
performing schools 

Having authority to intervene in 
the lowest-achieving schools and 
local education agencies 

No items in state interview aligned with this objective for ELLs b 


Turning around the lowest- 
achieving schools 

• Offer financial incentives for teachers with ELL expertise 

• Provide training on strategies for ELLs 

• Have state-level staff to support turnaround schools and 
districts in working with ELLs 

Encouraging conditions in 
which charter schools can 

Eliminating restrictions on charter 
school creation and enrollment 

No items in state interview aligned with this objective for ELLs b 

succeed 

Refining authorization and 
monitoring processes 

• Prioritize schools that address needs of ELLs 

• Monitor charter school performance based on the student 
populations (such as ELLs) served 

Source: RTT application; interviews with state administrators in spring 2012 and spring 2013. 
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ELL = English language learner. 

a Regular school districts are the most common type of local education agency. 

b The number of questions included in the state interview was purposefully limited to reduce the time it took to complete the 
interview. We initially developed the interview questions based on an examination of the RTT application criteria. To ensure that the 
interview was of a reasonable length, we then pared down the initial list of questions through a deliberative process with the Institute 
of Education Sciences and the RTT Program Office, to assess their priorities for the types of questions to include. The interview did 
not include any questions about this objective. 

• National Assessment of Education Progress (NAEP) scores. To examine the link between 
test scores and RTT, we obtained publicly available data from NAEP, a nationally 
representative assessment of U.S. students. We obtained state-level mean NAEP scores for 
the full NAEP sample as well as for the following subgroups: males, females, ELL students, 
Hispanic students, Asian students, black students, Native American students, white students, 
and students eligible for free or reduced-price lunch. The NAEP scores were available for 
grades 4 and 8, for both math and reading, every other year. In 2001, participation in state 
NAEP tests was made mandatory for states receiving Title I funds. Therefore, for early RTT 
states, we used four years of data before RTT grants (2003, 2005, 2007, and 2009) and three 
years of post-RTT data (2011, 2013, and 2015). For later RTT states, we used five years of 
pre-RTT data (2003, 2005, 2007, 2009, 2011), and two years of post-RTT data (2013 and 
2015). For all states, we also used NAEP scores to examine baseline differences in states’ 
achievement prior to RTT. 

We chose NAEP scores as our primary outcome for examining the link between RTT and 
student outcomes because they were measured consistently across all states over the time 
frame our study covered. Other outcomes, such as scores on state assessments, high school 
graduation rates, and college enrollment rates, were not. For example, many states changed 
their state assessments in the years following RTT awards. Many states did not measure high 
school graduation rates consistently. Few states collected college enrollment rates. We did 
collect data on both high school graduation rates and college enrollment rates but were 
unable to use them due to these limitations. The NAEP scores have two key limitations: (1) 
they were available for all states for only four years in the pre-RTT period, which made it 
difficult to accurately determine the pattern of outcomes in the pre-RTT period, and (2) the 
composition of students who took NAEP tests within each state varied from year to year, 
which meant that differences in outcomes due to changes in student characteristics could not 
be disentangled from differences in outcomes due to RTT. However, scores on state 
assessments also suffer from this second limitation, since they are also repeated cross- 
sectional samples for each particular grade, as opposed to longitudinal samples. 

• Common Core of Data (CCD). For baseline data on student and school characteristics and 
outcome data on state-level per-pupil spending, we obtained publicly available data from the 
CCD, which includes annual data about each public school, LEA, and state in the country. 
We obtained 2007-2008 state-level characteristics such as the percentage of students in each 
race or ethnicity category, the percentage of students who are ELLs, the percentage of 
students eligible for free or reduced-price lunch (an indicator of economic need), and the 
percentage of schools eligible for Title I (a program that provides financial assistance to 
LEAs with at least 10 children from low-income families and with at least 2 percent of the 
LEA's school-age population from low-income families). We also obtained state-level per- 
pupil spending for each year from 2004 to 2012. For early RTT states, we used seven years 
of pre-RTT data on per-pupil spending (2004 to 2010) and two years of post-RTT data 
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(2011 and 2012). For later RTT states, we used eight years of pre-RTT data on per-pupil 
spending (2004 to 201 1) and one year of post-RTT data (2012). 

B. Analysis methods 

In this section, we describe the methods we used to compare the policies and practices 
reported by RTT and other states, and to compare the student achievement outcomes experienced 
by these two groups of states. The purpose of these comparisons is not to detennine whether 
receipt of an RTT grant caused states to use particular policies and practices or to experience 
changes in student outcomes, but simply to determine: 

1 . Whether states that received an RTT grant used the policies and practices promoted by RTT, 
and how that compares to other states; and 

2. Whether receipt of an RTT grant was related to improvement in student outcomes. 

In interpreting the results from the state interviews, please note the following caveats: (1) the 
findings are based on self-reported use of policies and practices, 19 (2) 2 out of 18 RTT objectives 
were not addressed by interview questions, (3) the application wording left it to the states to 
decide many of the details about how to implement particular policies and practices, and (4) we 
did not collect infonnation about the quality or fidelity with which the policies and practices 
were implemented. 20 Thus, readers should use caution when interpreting the results. 


19 The study team conducted pilot tests of the interview instrument and provided training to the data collection team 
to ensure the uniformity and consistency of the data collected. The interview consisted of mostly closed-ended 
questions — that is, questions with yes or no responses or with a set of specific response categories from which to 
choose. The study team carefully reviewed all responses for completeness and consistency, and followed up with 
respondents about missing and inconsistent responses to ensure their accuracy. 

20 Here we provide several potential explanations for how these limitations might affect the data. Our data do not 
allow us to determine whether any of these explanations are correct, but we offer them here as starting points for 
thinking about how the results might be affected by the limitations. Self-reported levels of policy and practice use 
might be overestimated (relative to actual use) if states provided socially desirable responses, which would likely 
lead to overestimated levels for all states. The study team took several steps to ensure that states provided accurate 
responses, including telling states that the interview was not an audit and that we would report aggregated responses 
across states, rather than singling out any individual state. The fact that self-reported levels of policy and practice 
use (as presented in Chapter IV) are not all 100 percent, and in many cases are much lower than 100 percent, 
suggests that the issue of providing socially desirable responses is not a rampant problem in our data. However, it is 
possible that RTT states might be more likely than non-RTT states to provide socially desirable responses, given 
that they received RTT grants to implement the policies and practices we examined. Therefore, the results for RTT 
states might be more inflated than the results for non-RTT states, so readers should use caution when interpreting the 
results. Regarding the fact that our study instruments did not address 2 out of 18 RTT objectives, this factor is 
unlikely to have a large effect on the overall results because very few objectives were not addressed. Regarding the 
quality or fidelity with which the policies and practices were implemented, and the fact that the application wording 
left it to the states to decide many of the details about how to implement particular policies and practices, our data 
might overestimate use levels if states tended to report that they used a policy or practice when they had begun to 
use it, but hadn’t necessarily implemented it fully. This factor would lead to overestimated levels for all states, rather 
than affecting the differences between RTT and non-RTT states. In addition, as noted above, the fact that many of 
the self-reported levels of policy and practice use (as presented in Chapter IV) are well below 100 percent suggests 
that these levels might not be overestimated by a substantial amount. 
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We first describe how we formed the groups that are the basis for the comparisons presented 
in the report. We then describe how we summarized the large number of findings from state 
interviews, how we analyzed the extent to which states focused on ELLs in their use of policies 
and practices promoted by RTT, and how we examined the relationship between receipt of an 
RTT grant and student achievement. 

1. RTT comparisons 

We conducted two sets of RTT comparisons: (1) Round 1 and 2 RTT states (tenned early 
RTT states in this report) with states that did not receive RTT grants (termed non-RTT states or 
other states in this report) and (2) Round 3 RTT states (tenned later RTT states in this report) 
with non-RTT states. 21 We distinguish between Rounds 1 or 2 and Round 3 because of 
differences in the grants’ timing, funding levels, and scope for these groups of states (described 
in Chapter I). We combined Round 1 and 2 RTT states for several reasons: the funding amounts 
and scope of the grants in those rounds were similar, only two states received Round 1 RTT 
grants, and the first two rounds of grants were awarded within roughly five months of each other. 

When the interview data for this report were collected in spring 2012 and 2013, Round 1 and 
2 states had been implementing their grants substantially longer than Round 3 states 
(approximately 20 to 37 months for Round 1 and 2 states versus approximately 4 to 16 months 
for Round 3 states). In addition, less funding was available for Round 3 states, so they were 
asked to focus on a subset of areas. All seven Round 3 states focused on the areas of state 
capacity and standards and assessments, six focused on data systems and teacher and principal 
certification and evaluation, two focused on charter schools, and one focused on school 
turnaround. For these reasons, one might expect different progress from Round 3 states than 
from Round 1 and 2 states, both in terms of their use of RTT-promoted policies and practices and 
in tenns of student outcomes. 

2. Summarizing findings from the state interviews 

Given the large number of questions in the interviews, it is difficult to discern broad patterns 
or form overall conclusions by only separately examining responses to individual questions. 
Therefore, we analyzed data from the interviews using methods (described below) designed to 
provide infonnation about broad patterns observed in the data. Readers interested in the 
responses to specific interview questions can refer to Appendix B. 

Examining use of policies and practices in spring 2013. To summarize the large amount 
of data collected, we identified state interview questions that aligned with the policies and 
practices that RTT sought to affect. Throughout the report, we use the tenn “RTT-promoted 
policies and practices” to mean “policies and practices aligned with the RTT application 
criteria.” We determined how many policies and practices each state reported using and then 
calculated the average number of policies and practices for early RTT states, later RTT states, 
and non-RTT states. We then tested whether differences were statistically significant between 
each of the RTT groups and the non-RTT group in the average number of policies and practices 
reported. Throughout the report, when we say that one group of states (such as early RTT states) 
reported using more/less practices than another group (such as non-RTT states) within a 


21 We did not compare Round 1 and Round 2 states to Round 3 states because this report is focused on comparing 
states that received RTT grants to those that did not. 
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particular topic or subtopic area, we are always reporting findings that were statistically 
significant. In contrast, we did not conduct statistical tests to assess whether differences in 
individual practices between RTT and other states were statistically significant. See Appendix A 
for more details on the analysis methods. 

Examining use of policies and practices prior to RTT awards. In addition to examining 
the differences in policies and practices for all six areas in spring 2013, for three areas (teacher 
and principal certification and evaluation, school turnaround, charter schools) we examined 
whether states’ use of policies and practices differed during the 2007-2008 school year. This was 
possible because the spring 2012 interviews collected data not only on policies and practices 
used during the 2011-2012 school year but also baseline data about selected policies and 
practices in these areas during 2007-2008 before the announcement and implementation of 
RTT. 22 For the other three areas, it was not possible to examine baseline data because the 
interview did not include any questions about the 2007-2008 school year. 23 We focused the 
baseline data collection on the 2007-2008 school year because after the American Recovery and 
Reinvestment Act was announced in February 2009, states may have started taking steps to 
increase the competitiveness of their RTT applications in the focus areas described in the law. 

The policies and practices for which we could measure use during the 2007-2008 school 
year are a subset of those for which we could measure use in the 2011-2012 and 2012-2013 
school years. In Chapter IV, we list the policies and practices for which we could measure use in 
2011-2012 and 2012-2013 and indicate those for which we could also measure use in 2007- 
2008. It is important to keep this factor in mind when interpreting the baseline results. For 
example, finding no statistically significant difference at baseline between early RTT and non- 
RTT states for a particular area means that baseline differences did not exist for some of the 
policies and practices in that area, but we do not kn ow whether that was the case for all of the 
policies and practices in that area because we did not measure use of all policies and practices at 
baseline. 

Examining changes over time in use of policies and practices. We examined changes 
over time in two ways (findings presented in Appendix D): 


22 Interview responses to questions pertaining to earlier school years (such as 2007-2008) might suffer from recall 
error. That is, responses about earlier school years might differ from those about the current year simply because of 
the respondent’s inability to recall accurately the earlier time period rather than because of actual changes from year 
to year. This error could be systematic, resulting in bias, or random, resulting in less precise estimates. To address 
this concern, we sent our interview protocols to each respondent prior to the interview. In addition to drawing 
attention to items that required respondents to look up information prior to the interview, we directed their attention 
to questions that covered the 2007-2008 school year to make sure that they were prepared to answer those questions 
during the interview. For example, they may have needed to obtain the information about that period from a 
colleague or ask that a colleague with more knowledge of that time period also participate in the interview. We pilot 
tested this process to assess the extent to which respondents were able to report information on the 2007-2008 
school year, specifically asking whether respondents found it difficult or had any problems recalling that period. All 
respondents reported that they were able to accurately answer questions about that time period. Further, interviewers 
reported that respondents who were not in their current position during the earlier time period tended to ask 
colleagues who were knowledgeable about policies or practices at the earlier time to join the interview or to provide 
responses prior to the interview. 

23 When developing the interview protocol, we included baseline questions for particular policies and practices 
based on U.S. Department of Education guidance about those in which it had interest in assessing change. 
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1. Comparing use of policies and practices in 2011-2012 and 2012-2013. For all six areas, 
we examined whether use of RTT-promoted policies and practices changed between 20 11 — 
2012 and 2012-2013 and whether those changes differed between groups of states (early 
RTT, later RTT, non-RTT). To conduct this analysis, we averaged the number of policies 
and practices within each group of states, separately for each school year. We then 
subtracted the mean number of policies and practices for 2011-2012 from the mean number 
for 2012-2013, and tested whether the resulting number differed between early RTT and 
non-RTT states, and between later RTT and non-RTT states. In other words, this analysis 
tested whether the changes over time in the number of policies and practices used differed 
between RTT and other states. 

2. Comparing use of policies and practices in 2007-2008 and 2012-2013. Finally, for the 
three areas for which we had baseline data, we examined whether use of RTT-promoted 
policies and practices changed between 2007-2008 and 2012-2013 and whether those 
changes differed between the three group of states. This analysis includes only the subset of 
policies and practices for which we measured use in both 2007-2008 and 2012-2013. 
Specifically, we averaged the number of those policies and practices within each group of 
states, separately for each school year. We then subtracted the mean number of policies and 
practices for 2007-2008 from the mean number for 2012-2013, and tested whether the 
resulting number differed between early RTT and non-RTT states, and between later RTT 
and non-RTT states. In other words, this analysis tested whether the changes between 2007- 
2008 and 2012-2013 in the number of policies and practices used differed between RTT and 
other states. The results do not provide infonnation on whether there was a change over time 
for the policies and practices for which we did not measure use in 2007-2008. 

3. ELL-focused analyses of the state interviews 

We examined the extent to which states focused on ELLs in their use of RTT-promoted 
policies and practices using the same processes that we described above to analyze data from the 
state interviews; however, the summary measures included only policies and practices that 
explicitly focused on ELLs. We also examined whether use of these ELL-focused policies and 
practices differed by the size of the ELL population and the ELL/non-ELL achievement gap 
(which we refer to in this report as the ELL achievement gap for simplicity). We took the 
following steps to conduct these analyses: 

• We first identified ELL-focused state interview questions that aligned with the policies and 
practices that RTT sought to affect. We then determined how many ELL-focused policies 
and practices each state reported using. 

• We then categorized each state according to whether it had an above-median or below- 
median ELL population and an above- or below-median ELL achievement gap, where ELL 
population is defined as the percentage of students who are ELLs. We classified states as 
having higher (above-median) or lower (below-median) ELL populations using CCD data 
on the percentage of public school students participating in programs for ELLs in 2007- 
2008. We classified states as having higher or lower ELL achievement gaps based on their 
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gaps on the spring 2007 NAEP 4th grade math exam. 24 Specifically, we calculated the ELL 
achievement gap as average achievement for non-ELLs minus average achievement for 
ELLs. We used CCD data from 2007-2008 and NAEP data from spring 2007 because in 
later years, the American Recovery and Reinvestment Act had been announced, and it may 
have motivated states to start taking steps to increase the competitiveness of their RTT 
applications, possibly affecting their ELL population or ELL achievement gap. 

• We then examined the use of ELL-focused policies and practices for each of those groups 
(above-median ELL population, below-median ELL population, above-median gap, below- 
median gap). Throughout this report, we use “states with higher ELL populations” to refer to 
states with above-median ELL populations, “states with higher ELL achievement gaps” to 
refer to states with above-median ELL achievement gaps, “states with lower ELL 
populations” to refer to states with below-median ELL populations, and “states with lower 
ELL achievement gaps” to refer to states with below-median ELL achievement gaps. 

• Finally, we conducted two types of statistical tests: 

The first to determine whether there were differences in the number of ELL-focused 
policies and practices used between each of the RTT groups and non-RTT states. For 
example, we compared early RTT states with lower ELL populations and ELL 
achievement gaps to non-RTT states that also had lower populations and gaps. 

The second to determine whether there were differences in the number of ELL-focused 
policies and practices within each group of states. For example, we compared early RTT 
states with lower populations and gaps to early RTT states with higher populations and 
gaps. 

4. Examining the relationship between receipt of an RTT grant and student 
outcomes 

To examine the relationship between receipt of an RTT grant and student outcomes, we 
conducted a descriptive analysis of student outcomes before and after the award of RTT grants 
by plotting outcomes over time for the three groups of states described above (early RTT states, 
later RTT states, and non-RTT states). For each group, we plotted the average outcome across all 
states in that group for each year. 

Recognizing that RTT represents a significant investment in education reform, we carefully 
considered, in consultation with our expert panel, 25 whether it was possible to credibly estimate 
the effect of RTT on student outcomes using a comparative short interrupted time series design. 
This design compared how actual post-RTT outcomes in RTT and other states differed from 


24 We focused on the NAEP 4th grade math exam because compared with NAEP exams for other grades and 
subjects, the 4th grade math exam had the fewest states with a missing value for this achievement gap in 2007. The 
ELL achievement gaps on these exams were all highly correlated (the correlations ranged from 0.7 to 0.9), so the 
choice of subject and grade was unlikely to make a large difference in the composition of the lower and higher 
groups. 

25 Our expert panel consisted of experts in school reform and program evaluation. The group met three times during 
the evaluation, and also provided input through conference calls throughout the evaluation. 
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projected outcomes based on pre-RTT patterns (Somers et al. 2013). We tried this approach and 
decided not to present any results from it, for two reasons: 

1 . When we used this approach, we found that conclusions about the effect of RTT differed 
depending on what assumptions we made about the pattern of outcomes in the pre-RTT 
period, and it was difficult to detennine the true pattern of outcomes during that period 
because NAEP data were available for all states for only four years in that period; 26 and 

2. Even if it were possible to determine the true pattern of outcomes during the pre-RTT 
period, we could not conclude that receipt of an RTT grant caused any observed changes in 
student outcomes because other changes taking place at the same time as RTT reforms 
might also have affected student outcomes. 27 

To avoid misinterpretation of results that could result from using the interrupted time series 
method, we conducted a purely descriptive analysis. Appendix E presents these results. 


26 In particular, we tried fitting three different models to the pre-RTT data: a mean model (i.e., a flat line), a linear 
model, and a quadratic model. The estimates of the effect of RTT on student outcomes differed substantially 
depending on which model was used for the pre-RTT period, so we ultimately concluded — in consultation with 1ES 
and our expert panel members — that we could not draw any definitive conclusions from this approach. 

27 We also carefully considered, in consultation with our expert panel, using two other approaches: (1) a regression 
discontinuity design, and (2) a difference-in-differences approach with only one year of pre-RTT data. A regression 
discontinuity design would compare outcomes from states with values above the RTT application score cutoff (the 
treatment group) to outcomes from states with values below that cutoff (the comparison group). We decided not to 
use this approach for two reasons: (1) the number of states above the cutoff was too small to support reliable 
estimation of the relationship between the outcome and the application score and (2) the cutoff changed over time as 
more states were awarded RTT grants, exacerbating the small sample size problem. A difference-in-differences 
approach with only one year of pre-RTT data would compare outcomes for RTT and other states before and after the 
award of RTT grants. We decided not to use this approach because it suffers from the same weaknesses listed above 
for the comparative short interrupted time series design. 
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As described in Chapter I, RTT sought to promote and facilitate systemic changes at all 
levels of the U.S. educational system by supporting policies in six broad areas: (1) state capacity 
for reform; (2) rigorous standards and aligned assessments; (3) comprehensive data systems; (4) 
effective teachers and principals; (5) turnaround of low-performing schools; and (6) charter 
schools. The RTT grant competition, which was open to all states, the District of Columbia, and 
Puerto Rico, was designed to encourage and reward states that demonstrated past improvements 
in student achievement and presented comprehensive plans in their applications for 
implementing reforms in the areas outlined in the application. Only four states (Alaska, North 
Dakota, Texas, Vermont) did not compete for RTT in any round, and it is well documented that 
after RTT was announced, many states took steps to become eligible for the grants and to make 
their applications more competitive. 28 Therefore, when examining RTT implementation and the 
relationship between receipt of an RTT grant and student achievement, it is essential to 
understand the education policies and practices that existed in RTT states and other states before 
the RTT program was announced. 29 This is important to ensure that we do not draw misleading 
conclusions from our study’s findings. For example, any differences found between RTT and 
other states in spring 2013 could be due to the effect of RTT, but they also could reflect 
preexisting differences between these groups of states. 

In this chapter, we compare RTT states and other states to examine their similarities and 
differences prior to RTT. Baseline characteristics of the three groups of states — (1) early RTT 
states, (2) later RTT states, and (3) non-RTT states — are shown in Table III. 1. As discussed in 
Chapter II, the RTT analyses in this report were based on comparisons between groups (1) and 
(3) and between groups (2) and (3). Therefore, in the text that follows, we focus on contrasts 
between these groups of states. 


28 For instance, Illinois lifted caps on the number of charter schools it would allow, and Massachusetts made it easier 
for students in low-performing schools to switch to charters (National Alliance for Public Charter Schools, 2009); 
West Virginia proposed a merit pay system that included student achievement in compensation calculations 
(Council of State Governments 2010, 2011). To be eligible to apply for RTT, states could not have laws prohibiting 
the linking of student achievement and teacher administrative records, and some states (for example, California and 
Maine) changed their laws to be eligible (Government Accountability Office 2011). Most states also adopted the 
Common Core State Standards. 

29 RTT was first announced in July 2009, with the first round of state grant applications due to ED in January 2010. 
Therefore, for most of the state characteristics examined, we focus on information from the 2007-2008 school year. 
One exception to this rule is that we examine whether states had received any State Longitudinal Data Systems 
(SLDS) Grants between 2005 and 2009, and the cumulative amount of funding received during this period. Because 
states would have had to apply for these grants prior to July 2009, this grant funding would have reflected state 
efforts to improve their longitudinal data systems before RTT was first announced. Another state characteristic of 
interest is the number of states that had Elementary and Secondary Education Act flexibility waivers, which 
promoted similar policies as RTT. States applied for these waivers after August 201 1, so information on the number 
of states that had waivers is not included in Table 111. 1 , which focuses on information from 2007-2008 (prior to 
RTT). However, RTT could have affected the likelihood that states received these waivers; 91.7 percent of early 
RTT states and 85.7 percent of later RTT states received these waivers for the 2012-2013 school year (the first 
school year in which the waivers began), compared to 59.4 percent of non-RTT states. 
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Table 111.1. Baseline characteristics of early RTT states, later RTT states, 
and non-RTT states 



Early RTT 
states 

Later RTT 
states 

Non-RTT 

states 

RTT Round 1 Application Scores, Among States That Applied in Round V 

Average RTT Round 1 application scores 

419* 

388* 

323 

Average RTT Round 1 application scores on state reform 
conditions criteria 

218* 

205* 

173 

Average RTT Round 1 application scores on reform plan 
criteria 

200* 

184* 

149 

National Assessment of Educational Progress (NAEP) Scores, 2007 

Average scores: math, 4th grade 

238 

238 

240 

Average scores: math, 8th grade 

278 

281 

282 

Average scores: reading, 4th grade 

220 

220 

221 

Average scores: reading, 8th grade 

260 

262 

263 


State Longitudinal Data System (SLDS) Grant Information 


Percentage of states that received SLDS grants in 2005, 2007, 
or 2009 

Average number of SLDS grants received from 2005 to 2009 
among states that did and did not receive these grants' 3 
Average total SLDS funding received from 2005 to 2009 
among states that did and did not receive these grants 

91.7 

1.2 

$5.4 million 

85.7 

1.1 

$6.0 million 

78.1 

1.0 

$4.8 million 

State Fiscal Stabilization Fund (SFSF) Grant Information 

Percentage of states that received SFSF grants in 2009 

100 

100 

100 

Average 2009 SFSF grant amount 

$1.16 billion 

$1.20 billion 

$0.79 billion 

State-Reported Usage of Policies and Practices, from Interview Responses (2007-2008) c 


Average number of RTT-promoted policies and practices in the 
area of teacher and principal certification and evaluation (out 

of 12) 

4.6* 

4.5* 

2.7 

Average number of RTT-promoted policies and practices in the 

area of school turnaround (out of 5) 

1.3 

1.6* 

0.8 

Average number of RTT-promoted policies and practices in the 

area of charter schools (out of 3) 

2.0 

2.1 

1.3 

State Education Characteristics, from the Common Core of Data (2007-2008) 



Student’s race/ethnicity (average percentage) 

White, non-Hispanic 

50.9* 

60.6 

69.2 

Black, non-Hispanic 

26.6* 

16.9 

11.1 

Hispanic 

11.6 

17.3 

12.3 

Asian 

9.2 

3.4 

3.1 

Other 

1.7 

1.9 

4.3 

Average percentage of students eligible for free or reduced- 

price lunch 

40.6 

40.4 

39.9 

Average percentage of schools eligible for Title 1 

68.8 

70.4 

63.4 

State’s region (percentage) 

Northeast 

25.0 

28.6 

12.5 

Midwest 

8.3 

14.3 

31.3 

South 

58.3* 

28.6 

25.0 

West 

8.3 

28.6 

31.3 

School level (average percentage) 

Elementary 

61.0 

61.2 

57.8 

Middle 

18.8 

18.0 

18.0 

High 

15.9* 

17.4 

19.6 
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Early RTT 

Later RTT 

Non-RTT 


states 

states 

states 

Other 

4.4 

3.5 

4.6 

Average number of full-time staff 

153,000 

153,000 

104,000 

Number of States 

11-12 

7 

23-32 


Sources: U.S. Department of Education; NAEP scores; interviews with state administrators in spring 2012; Common 
Core of Data. 

Note: The characteristics reported in this table are unweighted state averages. A range is provided for the sample 

sizes because nonresponse varied across items. The results for non-RTT states shown in this table differ 
slightly from the results for non-RTT states shown in Table 111.1 of an earlier report from this evaluation 
(Dragoset et al. 2015) because Texas did not participate in the spring 2012 interviews and was therefore 
excluded from Table 111.1 in the earlier report. 

a Alaska, North Dakota, Texas, and Vermont did not apply for RTT grants in any round and are excluded from this 
section of the table because they have no application scores. 

b States were able to receive more than one SLDS grant across the three rounds of SLDS awards in 2005, 2007, and 
2009. 

c We identified the spring 2012 state interview questions that asked about RTT-promoted policies and practices used 
during 2007-2008. A higher number indicates that the state reported using more policies and practices promoted by 
RTT in school year 2007-2008. All data provided by states were self-reported and not independently verified by the 
research team. For these reasons and potential concerns about recall accuracy, readers should exercise caution 
when interpreting data from 2007-2008. Texas did not participate in the spring 2012 interviews and is excluded from 
this section of the table. 

d This value represents the average number of full-time equivalent staff; a staff member who works part time is 
counted as a proportion of, rather than a whole, staff member. For example, two half-time employees are counted as 
one full-time equivalent staff member. The average numbers of full-time equivalent staff were rounded to the nearest 
thousand. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 

The groups of states differed by a statistically significant margin at baseline on the following 
characteristics: 

• RTT Round 1 application scores. As one might expect, the average total scores received 
from grant application reviewers for the Round 1 applications of both early and later RTT 
states (419 and 388) were higher than the average scores for non-RTT states that applied for 
but did not win RTT grants (323). 30 A maximum score of 500 points was possible (see U.S. 
Department of Education [2016a] for the scoring rubric). More than half of the points that a 
state’s application could receive (260) were based on demonstrated accomplishments prior 
to applying — for example, increasing student achievement, enlisting statewide support for 
proposed plans, or creating legal conditions conducive to policy changes. Compared to non- 
RTT states that applied for RTT, both early and later RTT states had higher scores on these 
accomplishments (referred to as “state refonn conditions” in the application criteria) prior to 
applying. The higher scores suggest potentially important preexisting differences between 
these groups of states. The balance of points that a state’s application could receive was 
awarded based on the state’s proposed policy plans. Both early and later RTT states had 
higher scores than non-RTT states that applied for RTT on these plans (referred to as “state 
reform plans” in the application criteria). 


30 We focus on Round 1 scores to use scores from the greatest number of applicants. We would be missing 1 1 states 
if we used Round 2 scores (i.e., the 2 winning states from Round 1 and the 9 states that applied in Round 1 only but 
did not win a grant), whereas using Round 1 scores eliminates 6 states that only applied in Round 2. 
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• Policies and practices for teacher and principal certification and evaluation and for 
school turnaround. Both early and later RTT states reported using more RTT-promoted 
policies and practices in the 2007-2008 baseline year in the area of teacher and principal 
certification and evaluation (4.6 for early RTT states, 4.5 for later RTT states, 2.7 for non- 
RTT states). Later RTT states reported using more RTT-promoted policies and practices in 
the 2007-2008 baseline year in the school turnaround area (1.6 for later RTT states, 0.8 for 
non-RTT states). 

• State education characteristics. Compared to non-RTT states at baseline, according to the 
Common Core of Data for 2007-2008, early RTT states had a higher percentage of students 
who were black (26.6 vs. 11.1 percent), a lower percentage of students who were white 
(50.9 vs. 69.2 percent), were more likely to be located in the south (58.3 vs. 25.0 percent), 
and had a smaller percentage of high schools (15.9 vs. 19.6 percent). 

The groups of states did not differ to a statistically significant degree on the following 

baseline characteristics: 

• Average 2007 NAEP scores for students in 4th grade and 8th grade in both reading 
and mathematics. 

• Percentage of states that received at least one SLDS grant between 2005 and 2009, 
average number of SLDS grants received over that time period, and average amount of 
total SLDS grant funding received over that time period. 31 

• Percentage of states that received an SFSF grant and average amount of SFSF grant 
funding received. 32 

• Average number of RTT-promoted policies and practices in the 2007-2008 baseline 
year in the area of charter schools. 

• State education characteristics. Later RTT states and non-RTT states did not differ by a 
statistically significant margin on any of the variables examined from the Common Core of 
Data for 2007-2008. Early RTT states and non-RTT states did not differ by a statistically 
significant margin in terms of the percentage of students who were Hispanic, Asian, or other 
race; the percentage of students eligible for free or reduced-price lunch; the percentage of 
schools eligible for Title I; the percentage of states from the Northeast, Midwest, and West; 
the percentage of elementary, middle, and other schools; or the number of full-time staff. 


31 States were able to receive more than one SLDS grant across the three rounds of SLDS awards in 2005, 2007, and 
2009. 

32 It is important to understand whether RTT and other states differed in their receipt of SFSF grants and the amount 
of SFSF grant funding received because the SFSF grants were large (the average 2009 SFSF grant amount for early 
RTT states was $1.16 billion) and were awarded to governors in exchange for a commitment to advance many of the 
same types of education reforms that were promoted by RTT, such as college- and career-ready standards and high- 
quality, valid, and reliable assessments for all students; development and use of pre-K through post-secondary and 
career data systems; increasing teacher effectiveness and ensuring an equitable distribution of qualified teachers; and 
turning around the lowest-performing schools. 
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These results suggest that some important baseline differences existed between the groups of 
states before RTT efforts got underway. When interpreting this study’s RTT findings, it will be 
important to keep these differences in mind. Any differences in policies and practices or student 
achievement in post-RTT years may not necessarily be attributable to the RTT program and 
instead may reflect pre-existing differences between the groups. 
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IV. STATES’ USE OF POLICIES AND PRACTICES PROMOTED BY RACE TO THE 
TOP 


To understand the ultimate effectiveness of a grant program like RTT in improving student 
achievement, it is important first to understand the extent to which the policies and practices it 
promotes are being used by states. If none (or very few) of the RTT states used these policies and 
practices, it is unlikely that any changes in outcomes — positive or negative — could be attributed 
to the program. In this chapter, we assess the extent to which RTT and other states reported using 
policies and practices promoted by RTT. 

As noted previously, RTT promoted specific policies in six topic areas. In this chapter, we 
summarize the extent to which state administrators reported in spring 2013 using RTT-promoted 
policies and practices in each of these topic areas and their subtopics. Appendix A presents more 
detailed findings for the subtopics. Appendix D presents changes over time in the use of these 
policies and practices. 

A. No significant differences between RTT and other states in use of 

practices in the state capacity area 

A premise of RTT is that state education agencies can play a direct role in advancing 
statewide education policies. Thus, RTT aims to improve state capacity to support education 
policy. Section A of the RTT application, State Success Factors, focused on three subtopics: (1) 
articulating the state’s education reform agenda and LEAs’ participation in it; (2) building strong 
statewide capacity to implement, scale up, and sustain the proposed plans; and (3) demonstrating 
significant progress in raising achievement and closing achievement gaps. We collected data on 
the first two; no state interview questions aligned to the third subtopic, so it was excluded from 
the analysis. 33 The evaluation’s state interviews asked about 10 policies and practices aligned 
with RTT objectives in this topic area (Table IV. 1). 

In spring 2013, there were no statistically significant differences between RTT and 
other states in the state capacity area. Early RTT states, later RTT states, and non-RTT states 
reported using an average of 7. 1, 6.8, and 6.2 of the 10 RTT-promoted policies and practices in 
this area (Figure IV. 1). Results for the two subtopics are shown in Appendix A, Figures A.l and 
A.2. 


33 To limit the interview length, we sought input from IES and the RTT program office on which questions were of 
greatest interest. 
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Table IV.1. Policies and practices aligned with RTT objectives on state 
capacity, by subtopic 


Articulating the State’s education reform agenda and LEAs’ participation in It 


Having a comprehensive education reform plan in place 

Prioritizing the adoption and implementation of college- and career-ready standards or assessments 

Prioritizing the development of comprehensive, student-level, longitudinal data systems or using data to improve 
instruction 

Prioritizing the recruiting, developing, rewarding, and retaining of effective teachers and principals, especially 
where they are needed most (including developing and implementing evaluation systems) 

Prioritizing the turnaround of its lowest-achieving schools 


Building strong statewide capacity to implement, scale up, and sustain the proposed plans 


Providing leadership or teams to support the implementation of reform plans 

Working with districts that have the highest achievement levels and seeking to replicate their practices statewide to 
implement the education reform plan 

Providing effective and efficient operations and processes for grant oversight and performance measure tracking 
and reporting 

Involving teachers, teachers unions or associations, or school administrators in defining its education reform 
initiatives or priorities 

Involving other stakeholders in defining its education reform initiatives or priorities 


Source: Section A of the RTT application (U.S. Department of Education 2010); interviews with state administrators 
in spring 2012 and 2013. 

Note: The spring 2012 and spring 2013 state interviews asked whether these policies and practices were used 

during the current school year (either 201 1-2012 or 2012-2013). See Appendix C for a list of the specific 
interview questions that were aligned with the RTT policies and practices in this table. See the RTT 
application for a detailed description of each subtopic. 

LEA = local education agency. 
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Figure IV.1. Use of policies and practices aligned with RTT objectives on 
state capacity 



(N=12) (N=7) (N=32) 

Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV.1. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 1 0 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For example, two 
non-RTT states reported using 5 of the 10 state capacity policies and practices aligned with the RTT 
application criteria. For 3 of the policies and practices, a “yes” response received one point. In the other 7 
cases, it was possible for a state to receive a fraction of one point. See Appendix A for details on how we 
determined the number of policies and practices for each state. The dashed line denotes the average 
number of policies and practices for each group of states. There were no statistically significant differences 
between RTT and non-RTT states at the 0.05 level using a two-tailed test. 
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The two practices with the largest differences between early RTT and non-RTT states in the 
state capacity area were: (1) prioritizing school turnaround, with 100 percent of early RTT states 
and 78 percent of non-RTT states using this practice (Figure IV. 2), and (2) working with districts 
that have the highest achievement levels and seeking to replicate their practices statewide to 
implement the education reform plan (with 83 percent of early RTT states and 57 percent of non- 
RTT states using this practice) (Figure IV.3). 34,35 

The practice with the largest difference between later RTT and non-RTT states in this area 
was providing effective and efficient operations and processes for grant oversight and 
perfonnance measure tracking and reporting. On average, later RTT states used 8 1 percent of the 
components of this practice and non-RTT states used 62 percent (Figure IV.3). 

There were several practices in this area for which states reported using most of the 
components of the practice: (1) having a comprehensive education reform plan in place, (2) 
prioritizing the adoption and implementation of college- and career-ready standards or 
assessments, (3) prioritizing school turnaround, and (4) providing leadership or teams to support 
the implementation of reform plans (Figures IV.2 and IV.3). The practice in this area with the 
fewest components used was involving other stakeholders in defining education refonn 
initiatives or priorities (Figure IV.3). 


34 Descriptive findings on individual practices with the largest differences between RTT and other states are shown 
for readers who may be interested in specific practices where large differences were reported. We did not conduct 
statistical tests to assess whether differences in individual practices between RTT and other states were statistically 
significant. 

35 As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 
one or more relevant interview questions, we used those questions to calculate the percentage of questions to which 
the state responded “yes.” This variable measures the percentage of the components of the policy or practice that 
each state used. We then calculated the average percentage across all states. For example, suppose a particular 
practice was addressed by a single interview question. If half of the states used that practice and the other half did 
not, then we would say that 50 percent of states used that practice. Some policies and practices were based on 
multiple interview questions (rather than a single interview question), so each state could use less than 100 percent 
of the components of these policies and practices. For example, the application criteria specified that the state use 
support from a broad group of stakeholders to implement its education policy plans. The interview included eight 
separate questions about whether each of eight types of stakeholders (such as local school boards and community 
organizations) were involved in defining the state’s education policy plans. In this example, a state received 12.5 
percent, or one-eighth of 100 percent, for each “yes” response. If half of the states responded “yes” to all eight 
questions and the other half responded “yes” to none of the eight questions, we would say that states, on average, 
used 50 percent of the components of this practice. As demonstrated by these two examples, in the report, for 
practices that were addressed by a single interview question, we indicate the percentage of states that used each 
practice. For practices that were addressed by multiple interview questions, we indicate the average percentage of 
the components of each practice that states used. 
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Figure IV. 2. Use of policies and practices aligned with RTT, articulating 
state’s reform agenda subtopic 
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Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 

one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. 
The height of each bar represents the average percentage of the components of the policy or practice that 
each group of states used. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 
practice. 

CCR = college and career ready; SLDS = state longitudinal data system. 
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Figure IV. 3. Use of policies and practices aligned with RTT, building strong 
statewide capacity subtopic 
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I Early RTT States (N=12) | Later RTT States (N=7) | Non-RTT States (N=28-32) 

Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 

one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. 
The height of each bar represents the average percentage of the components of the policy or practice that 
each group of states used. A range is provided for the sample sizes because nonresponse varied across 
items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 
practice. 


B. Early RTT states reported using more standards and assessments 

practices than non-RTT states 

One goal of RTT is for states to use standards and assessments that prepare students to 
succeed in college and the workplace, and to compete in the global economy. The Standards and 
Assessments section of the RTT application (section B) focused on three subtopics: 

(1) developing and adopting common standards; (2) developing and implementing common, 
high-quality assessments; and (3) supporting the transition to enhanced standards and high- 
quality assessments. The study’s state interviews asked about 10 policies and practices aligned 
with RTT objectives in this topic area (Table IV. 2). 
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Table IV.2. Policies and practices aligned with RTT objectives on standards 
and assessments, by subtopic 


Developing and adopting common standards 


Adopting the Common Core State Standards (CCSS) in both English/language arts and math 


Developing and implementing common, high-quality assessments 


Participating in a consortium of states to develop assessments aligned to CCSS 

Specifying the school year by which state plans to fully implement summative assessments being developed by 
the consortium 


Supporting the transition to enhanced standards and high-quality assessments 


Supporting the implementation of CCSS by providing funds for additional staff or making new technology 
investments 

Supporting the implementation of the assessments associated with the CCSS by making new technology 
investments 

Requiring or supporting the use of new instructional materials for implementing the CCSS 

Developing, supporting, or requiring new interim assessments associated with the CCSS 

Changing high school exit requirements or college entrance requirements 

Changing policies about or providing funds for professional development, training, and technical assistance for 
teachers or school principals to support implementation of CCSS 

Supporting districts or schools in implementing the CCSS with English language learners 


Source: Section B of the RTT application (U.S. Department of Education 2010); interviews with state administrators 
in spring 2012 and 2013. 

Note: The spring 2012 and spring 2013 state interviews asked whether these policies and practices were used 

during the current school year (either 201 1-2012 or 2012-2013). See Appendix C for a list of interview 
questions that were aligned with the RTT policies and practices in this table. See the RTT application for a 
detailed description of each subtopic. 

In spring 2013, early RTT states reported using more RTT-promoted policies and 
practices in the standards and assessments area than non-RTT states. Early RTT states 
reported using an average of 7.2 out of 10 policies and practices in this area, compared to 4.6 for 
non-RTT states (Figure IV.4). Baseline data on policies and practices in this area were not 
collected, so we could not determine whether this difference existed before RTT grants were 
awarded. 
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Figure IV.4. Use of policies and practices aligned with RTT objectives on 
standards and assessments 



(N=12) (N=7) (N=32) 


Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 2. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 1 0 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For example, 
three early RTT states reported using 5 of the 10 standards and assessment policies and practices aligned 
with the RTT application criteria. For 5 of the policies and practices, a “yes” response received one point. In 
the other 5 cases, it was possible for a state to receive a fraction of one point. See Appendix A for details 
on how we determined the number of policies and practices for each state. The dashed line denotes the 
average number of policies and practices for each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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In spring 2013, for one of the three standards and assessment subtopics, early RTT 
states reported using more RTT-promoted policies and practices than non-RTT states. 

Results for the three subtopics are shown in Appendix A, Figures A. 3 through A. 5, and results 
for individual policies and practices are shown below in Figures IV. 5 and IV. 6. Early RTT states 
reported using an average of 4.3 out of 7 policies and practices in the subtopic focused on 
supporting the transition to enhanced standards and high-quality assessments, compared to 2.3 
for non-RTT states (Appendix A, Figure A. 5). Therefore, it may not be surprising to find that the 
practice with the largest difference between the two groups of states was a practice in this 
subtopic — supporting the implementation of the CCSS by providing funds for additional staff or 
making new technology investments. On average, early RTT states used 7 1 percent of the 
components of this practice and non-RTT states used 16 percent (Figure IV. 6). 

The practice with the largest difference between later RTT and non-RTT states in this area 
was supporting districts or schools in implementing the CCSS with English language learners. 
On average, a smaller percentage of later RTT states than non-RTT states used this practice (43 
versus 69 percent) (Figure IV. 6). 

There were some practices that most states reported using in this area: (1) adopting the 
CCSS in both English/language arts and math, (2) participating in a consortium of states to 
develop assessments aligned to CCSS, and (3) specifying the school year by which the state 
plans to fully implement summative assessments developed by the consortium in which the state 
participated (Figure IV. 5). In contrast, states reported using few of the components of the 
following practices: (1) developing, supporting, or requiring new interim assessments associated 
with the CCSS and (2) changing high school exit requirements or college entrance requirements 
(Figure IV. 6). 
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Figure IV.5. Use of policies and practices aligned with RTT, developing and 
adopting common standards subtopic and developing and implementing 
common, high-quality assessments subtopic 
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Developed by Consortium 


Policy or Practice Aligned with RTT Objectives 
I Early RTT States (N=12) | Later RTT States (N=6-7) | Non-RTT States (N=29-32) 

Source: Interviews with state administrators in spring 2013. 

Note: This figure has a separate panel for each subtopic. As described in Appendix A, for each policy or practice 

in the RTT application criteria for which we identified one or more interview questions aligned with the 
policy or practice, we calculated the percentage of interview questions with a “yes” response as a measure 
of the percentage of components each state used. The height of each bar represents the average 
percentage of the components of the policy or practice that each group of states used. A range is provided 
for the sample sizes because nonresponse varied across items. 

CCSS = Common Core State Standards. 
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Figure IV.6. Use of policies and practices aligned with RTT, supporting the 
transition to enhanced standards and high-quality assessments subtopic 
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Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 

one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. 
The height of each bar represents the average percentage of the components of the policy or practice that 
each group of states used. A range is provided for the sample sizes because nonresponse varied across 
items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 

practice. 

CCSS = Common Core State Standards; ELLs = English language learners; PD = professional development. 
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C. No significant differences between RTT and other states in use of 

practices in the data systems area 

One aim of RTT is for states to develop data systems that measure student achievement 
growth and success and to use those systems to infonn teachers’ and principals’ efforts to 
improve instruction. Section C of the RTT application, Data Systems to Improve Instruction, 
focused on three subtopics: (1) fully implementing an SLDS, (2) accessing state data and using it 
to inform key stakeholders, and (3) using data to improve instruction. The evaluation’s state 
interviews asked about eight policies and practices aligned with RTT objectives in this area 
(Table IV.3). 

Table IV.3. Policies and practices aligned with RTT objectives on data 
systems, by subtopic 


Fully implementing an SLDS 


Having an SLDS 

SLDS contains program participation information 3 
SLDS is linked to an early childhood data system 
SLDS is linked to a higher education data system 


Accessing state data and using it to inform key stakeholders 


SLDS can be accessed by stakeholders 

SLDS is used to inform and engage stakeholders and support decision makers in continuous improvement efforts 


Using data to improve instruction 


Requiring districts to implement district data systems' 5 

Providing funding, materials, training, technical assistance, or other supports to districts to encourage the use of 
data to improve instruction 

Source: Section C of the RTT application (U.S. Department of Education 2010); interviews with state administrators 
in spring 2012 and 2013. 

Note: The spring 2012 and spring 2013 state interviews asked whether these policies and practices were used 

during the current school year (either 201 1-2012 or 2012-2013). See Appendix C for a list of interview 
questions that were aligned with the RTT policies and practices in this table. See the RTT application for a 
detailed description of each subtopic. 

3 The interview questions comprising this policy or practice focused primarily on ELLs. See Appendix C for the 
interview questions that comprised each policy or practice. 

b District data systems, which also are called local instructional improvement systems, are defined by the 
U.S. Department of Education in the RTT application materials as technologically based tools or strategies that 
provide educators with data to manage continuous instructional improvement efforts. 

There were no differences in spring 2013 between RTT and other states in the data 
systems area. Early RTT states, later RTT states, and non-RTT states reported using averages of 
6.6, 6.6, and 5.8 of the 8 RTT-promoted policies and practices in this area (Figure IV. 7). Results 
for the three subtopics are shown in Appendix A, Figures A. 6 through A. 8. 
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Figure IV.7. Use of policies and practices aligned with RTT objectives on 
data systems 
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Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 3. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 
eight examined) that were aligned with the RTT application criteria. The number inside each dot is the 
number of states represented by the dot; dots that represent only one state have no number inside. For 
example, one non-RTT state reported using four of the eight data systems policies and practices aligned 
with the RTT application criteria. For five of the policies and practices, a “yes” response received one point. 
In the other three cases, it was possible for a state to receive a fraction of one point. See Appendix A for 
details on how we determined the number of policies and practices for each state. The dashed line denotes 
the average number of policies and practices for each group of states. There were no statistically significant 
differences between RTT and non-RTT states at the 0.05 level using a two-tailed test. 


The largest difference between early RTT and non-RTT states in this area was having an 
SLDS li nk ed to an early childhood data system. On average, 100 percent of early RTT states used 
this practice, compared with 65 percent of non-RTT states (Figure IV. 8). The two practices with 
the largest differences between later RTT and non-RTT states in this area were ( 1 ) having an SLDS 
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li nk ed to a higher education data system (with 100 percent of later RTT states and 75 percent of 
non-RTT states using this practice), and (2) using the SLDS to inform and engage stakeholders 
and support decision makers in continuous improvement efforts (with later RTT states and non- 
RTT states using, on average, 77 and 54 percent of the components of this practice) (Figure IV. 8). 
There were several practices that nearly all states reported using including: (1) having an SLDS, 
and (2) providing funding, materials, training, technical assistance, or other supports to districts to 
encourage the use of data to improve instruction (Figure IV. 8). 


Figure IV.8. Use of policies and practices aligned with RTT objectives on 
data systems 
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Source: Interviews with state administrators in spring 2013. 

Note: This figure has a separate panel for each subtopic. As described in Appendix A, for each policy or practice 

in the RTT application criteria for which we identified one or more interview questions aligned with the 
policy or practice, we calculated the percentage of interview questions with a “yes” response as a measure 
of the percentage of components each state used. The height of each bar represents the average 
percentage of the components of the policy or practice that each group of states used. A range is provided 
for the sample sizes because nonresponse varied across items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 

practice. 

EC = early childhood; HE = higher education; SLDS = statewide longitudinal data system. 
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D. RTT states reported using more teacher and principal certification and 
evaluation practices than other states 

Recruiting, developing, rewarding, and retaining effective teachers and principals, especially 
in high-poverty schools and districts where they are needed most, is one key objective of RTT. 
The teacher and principal certification and evaluation section of the RTT application (section D) 
placed specific emphasis on five subtopics: (1) providing high-quality pathways to certification 
for aspiring teachers and principals, (2) improving teacher and principal effectiveness based on 
perfonnance, (3) ensuring equitable distribution of effective teachers and principals, (4) 
improving the effectiveness of teacher and principal preparation programs, and (5) providing 
effective support to teachers and principals. We collected data on the first four subtopics, through 
state interview questions about 39 policies and practices aligned with RTT objectives in this area 
(Table IV.4). No state interview questions aligned to the fifth subtopic, so it was excluded from 
the analysis. 

RTT states reported using more RTT-promoted policies and practices in the teacher 
and principal certification and evaluation area in spring 2013 than other states. Early and 
later RTT states reported using 14.4 and 12.9 of the 39 policies and practices in this area, on 
average, compared to 7.8 for other states (Figure IV. 9). However, for both early and later RTT 
states, baseline data on 12 of these 39 policies and practices show that a difference between the 
RTT and other states existed prior to the awarding of RTT grants for some policies and practices. 
In particular, in 2007-2008, early and later RTT states reported, on average, using 4.6 and 4.5 of 
these 12 policies and practices, compared to 2.7 for other states. (Baseline data were not 
collected for the other 27 policies and practices, so we could not determine whether there were 
baseline differences for the topic area as a whole.) 

In spring 2013, for three of the four teacher and principal certification and evaluation 
subtopics, early RTT states reported using more RTT-promoted policies and practices than 
non-RTT states (results for the four subtopics are shown in Appendix A, Figures A. 9 through 
A. 12, and results for individual policies and practices are shown below in Figures IV. 10 through 
IV. 15): 

• Providing high-quality pathways to certification for aspiring teachers and principals. 

Early RTT states reported using an average of 6.0 out of 14 policies and practices in this 
area, compared to 4. 1 for non-RTT states (Appendix A, Figure A. 9). 

• Improving teacher and principal effectiveness based on performance. Similarly, early 
RTT states reported using 7.6 out of 15 policies and practices in this area on average, 
compared to 3.4 for non-RTT states (Appendix A, Figure A. 10). The practice with the 
largest difference between early RTT and non-RTT states — specifying a required minimum 
number of rating categories to be used when evaluating principals — fell in this subtopic. On 
average, 92 percent of early RTT states used this practice, compared with 43 percent of non- 
RTT states (Figure IV. 13). 

• Ensuring equitable distribution of effective teachers and principals. In this subtopic, 
early RTT states indicated that they used an average of 0.7 out of 4 policies and practices, 
relative to 0.2 for non-RTT states (Appendix A, Figure A. 1 1). 
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Table IV.4. Policies and practices aligned with RTT objectives on teacher 
and principal certification and evaluation 


Providing high-quality pathways to certification for aspiring teachers and principals 


Authorizing or expanding institutions qualified to operate alternative-route programs for teachers or principals 3 
Increasing the selectivity of alternative-route programs for teachers or principals 

Increasing the amount of mentoring for participants in certification programs or the time such participants spend in 
school-based learning experiences for teachers or principals 

Allowing alternative-route programs to award the same type of certification as traditional preparation programs for 
teachers or principals 

Operating alternative-route programs for teachers or principals 3 
Identifying areas of shortages for teachers or principals 
Addressing areas of shortage for teachers or principals 


Improving teacher and principal effectiveness based on performance 


Requiring districts to use student growth in evaluations and specifying the extent to which student achievement 
growth must factor into evaluations of teachers or principals 3 

Requiring multiple performance measures for evaluations of teachers or principals 3 

Specifying a required minimum number of rating categories to be used when evaluating teachers or principals 3 
Conducting annual evaluations of teachers 3b 

Requiring evaluations to inform decisions about professional development and support for teachers or principals 
Requiring evaluations to inform decisions about compensation for teachers or principals 
Requiring evaluations to inform decisions about career advancement for teachers or principals 
Requiring evaluations to inform decisions about dismissal of teachers or principals 


Ensuring equitable distribution of effective teachers and principals 


Requiring districts or schools to use strategies to promote a more equitable distribution of effective teachers or 
principals 

Analyzing effectiveness based on student achievement growth to determine whether the distribution of effective 
teachers or principals has shifted 


Improving the effectiveness of teacher and principal preparation programs 


Using effectiveness ratings (based in part on student achievement growth) to assess the effectiveness of 
certification programs for teachers or principals 3 

Publicly reporting results from evaluations of certification program effectiveness for teachers or principals 
Using results from evaluations of certification programs to provide additional funds for, expand, or promote 
certification programs that were shown to be effective for teachers or principals 


Source: Section D of the RTT application (U.S. Department of Education 2010); interviews with state administrators 
in spring 2012 and spring 2013. 

Note: The phrase “teachers or principals” indicates that we analyzed two separate practices: one focused on 

teachers and one focused on principals. The spring 2012 and 2013 state interviews asked whether these 
policies and practices were used during the current school year (either 201 1-2012 or 2012-2013). The 
spring 2012 state interview asked whether a subset of these policies and practices were used during the 
2007-2008 school year. See Appendix C for a list of interview questions that were aligned with the RTT 
policies and practices in this table. See the RTT application for a detailed description of each subtopic. 
Most of the rows in this table describe two separate practices (one for teachers, one for principals). 

3 The spring 2012 state interview asked whether this policy or practice was used during the 2007-2008 school year. 

b The state interview did not ask about this policy or practice for principals. 
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Figure IV.9. Use of policies and practices aligned with RTT objectives on 
teacher and principal certification and evaluation 



(N=12) (N=7) (N=32) 

Source: Interviews with state administrators in spring 201 3. 

Note: The policies and practices summarized in this figure are presented in Table IV. 4. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 39 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For example, two non- 
RTT states reported using slightly more than 15 of the 39 teacher and principal certification and evaluation 
policies and practices aligned with the RTT application criteria. For 25 of the policies and practices, a “yes” 
response received one point. In the other 14 cases, it was possible for a state to receive a fraction of one point. 
See Appendix A for details on how we determined the number of policies and practices for each state. The 
dashed line denotes the average number of policies and practices for each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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For one of the four teacher and principal certification and evaluation subtopics, later RTT 
states reported using more RTT-promoted policies and practices in spring 2013 than non-RTT 
states. In particular, later RTT states reported using an average of 7.4 out of 15 policies and practices 
in the subtopic focused on improving teacher and principal effectiveness based on perfonnance, 
compared to 3.4 for non-RTT states (Appendix A, Figure A. 10). Given the significant findings for 
this subtopic, perhaps it is not surprising that the two practices with the largest differences 
between later RTT and non-RTT states fell in this area. They were: (1) requiring evaluations to 
inform decisions about dismissal of teachers (with 71 percent of later RTT states and 22 percent 
of non-RTT states using this practice), and (2) requiring evaluations to inform decisions about 
dismissal of principals (with 57 percent of later RTT states and 10 percent of non-RTT states 
using this practice) (Figures IV. 12 and IV. 13). 

We found that nearly all states reported using one practice — identifying teacher shortage 
areas (Figure IV. 10). In contrast, there were several practices that no states reported using 
including: (1) using results from evaluations of certification programs to provide additional funds 
for, expand, or promote certification programs that were shown to be effective for teachers; (2) 
doing the same for certification programs that were shown to be effective for principals; and (3) 
publicly reporting results from evaluations of certification program effectiveness for principals 
(Figure IV. 15). 
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Figure IV.10. Use of teacher-focused policies and practices aligned with RTT, 
providing high-quality pathways to certification for aspiring teachers and 
principals subtopic 
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Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 

one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. 
The height of each bar represents the average percentage of the components of the policy or practice that 
each group of states used. A range is provided for the sample sizes because nonresponse varied across 
items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 
practice. 
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Figure IV.11. Use of principal-focused policies and practices aligned with 
RTT, providing high-quality pathways to certification for aspiring teachers 
and principals subtopic 
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Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 

one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. 
The height of each bar represents the average percentage of the components of the policy or practice that 
each group of states used. A range is provided for the sample sizes because nonresponse varied across 
items. The questions selected for the last practice in this figure (taking steps to address principal shortage 
areas) were only asked of states that reported having principal shortage areas. None of the later RTT 
states reported having a principal shortage area, so this group is missing a bar for this practice. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 
practice. 
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Figure IV.12. Use of teacher-focused policies and practices aligned with RTT, 
improving teacher and principal effectiveness based on performance 
subtopic 
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Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 

one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. 
The height of each bar represents the average percentage of the components of the policy or practice that 
each group of states used. A range is provided for the sample sizes because nonresponse varied across 
items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 

practice. 

PD = professional development. 
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Figure IV.13. Use of principal-focused policies and practices aligned with 
RTT, improving teacher and principal effectiveness based on performance 
subtopic 
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Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 

one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. 
The height of each bar represents the average percentage of the components of the policy or practice that 
each group of states used. A range is provided for the sample sizes because nonresponse varied across 
items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 

practice. 

PD = professional development. 
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Figure IV.14. Use of policies and practices aligned with RTT, ensuring 
equitable distribution of effective teachers and principals subtopic 


Ensuring Equitable Distribution of Effective Teachers and Principals 


0) 
eg 

55 

3k 
-Q 

1 
-D 
(0 

<D 70 

C 

E 
o 
O 

.8 50- 


90- 


80- 


60- 


a 

ra 


>. 

.y 

c 

CL 


40- 


30- 


20 - 


f 10-1 

8 

£ o 


ll 




Require Use of Strategies 
for More Equitable 
Distribution of 
Effective Teachers* 


Conduct Analyses of 
Shifts in Distribution of 
Effective Teachers 


Require Use of Strategies 
for More Equitable 
Distribution of 
Effective Principals* 

Policy or Practice Aligned with RTT Objectives 


Conduct Analyses of 
Shifts in Distribution of 
Effective Principals 


Early RTT States <N=12)H Later RTT States (N=7)H Non-RTT States (N=30-31) 


Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 

one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. 
The height of each bar represents the average percentage of the components of the policy or practice that 
each group of states used. A range is provided for the sample sizes because nonresponse varied across 
items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 
practice. 
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Figure IV.15. Use of policies and practices aligned with RTT, improving the 
effectiveness of teacher and principal preparation programs subtopic 
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Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each policy or practice in the RTT application criteria for which we identified 

one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. 
The height of each bar represents the average percentage of the components of the policy or practice that 
each group of states used. 
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E. Early RTT states reported using more school turnaround practices than 
non-RTT states 

Turning around the nation’s persistently lowest-achieving schools is another goal of RTT. 
Two subtopics were included in the turnaround section of the RTT application (section E): (1) 
authority to intervene in the lowest-achieving schools and LEAs, and (2) turning around the 
lowest-achieving schools. The evaluation’s state interviews asked about 10 policies and practices 
aligned with RTT objectives in this area (Table IV. 5). 

Table IV.5. Policies and practices aligned with RTT objectives on school 
turnaround, by subtopic 


Authority to intervene in the lowest-achieving schools and LEAs 


Having the authority to take over failing schools 3 


Turning around the lowest-achieving schools 


Providing training to the lowest-achieving schools or LEAs on analyzing student assessment data to improve 
instruction 

Helping the lowest-achieving schools or LEAs align curricula to state standards 

Providing training to the lowest-achieving schools or LEAs on identifying and implementing effective curricula, 
instructional strategies, or school intervention models, or developing and implementing a school improvement 
plan 

Providing training to the lowest-achieving schools or LEAs on identifying and implementing strategies to address 
the needs of ELLs 

Providing technical assistance to the lowest-achieving schools or LEAs on improving the quality of professional 
development 

Providing operational flexibility and support to lowest-achieving schools or LEAs with regard to staffing and 
budgeting 3 

Implementing or providing technical assistance on strategies to recruit and retain effective teachers (such as 
financial incentives) 

Having teacher tenure rules that affect placement in or removal from the lowest-achieving schools 3 

Having state-level administrative structures intended to support school turnaround efforts 3 

Source: Section E of the RTT application (U.S. Department of Education 2010); interviews with state administrators 
in spring 2012 and spring 2013. 

Note: The spring 2012 and spring 2013 state interviews asked whether these policies and practices were used 

during the current school year (either 201 1-2012 or 2012-2013). The spring 2012 state interview asked 
whether a subset of these policies and practices were used during the 2007-2008 school year. See 
Appendix C for a list of interview questions that were aligned with the RTT policies and practices in this 
table. See the RTT application fora detailed description of each subtopic. 

3 The spring 2012 state interview asked whether this policy or practice was used during the 2007-2008 school year. 
ELL = English language learner; LEA = local education agency. 

In spring 2013, early RTT states reported using more RTT-promoted policies and 
practices in the school turnaround area than non-RTT states. Early RTT states reported 
using an average of 6.5 of the 10 policies and practices in this area, compared to 5.3 for non-RTT 
states (Figure IV. 16). For 5 of these 10 policies and practices, no significant difference between 
early and non-RTT states existed before the awarding of RTT grants. (Baseline data were not 
collected for the other 5 policies and practices in this area, so we could not determine whether 
there were baseline differences for this topic area as a whole.) 
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Figure IV. 16. Use of policies and practices aligned with RTT objectives on 
school turnaround 



(N=12) (N=7) (N=32) 

Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV.5. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 1 0 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For example, 
three non-RTT states reported using 5 of the 10 school turnaround policies and practices aligned with the 
RTT application criteria. For 6 of the policies and practices, a “yes” response received one point. In the 
other 4 cases, it was possible for a state to receive a fraction of one point. See Appendix A for details on 
how we determined the number of policies and practices for each state. The dashed line denotes the 
average number of policies and practices for each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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For one of the two school turnaround subtopics, early RTT states reported using more 
RTT-promoted policies and practices than non-RTT states in spring 2013. Results for the 
two subtopics are shown in Appendix A, Figures A. 13 and A. 14, and results for individual 
policies and practices are shown below in Figure IV. 17. Early RTT states reported using an 
average of 6.2 out of 9 policies and practices in the subtopic focused on turning around the 
lowest-achieving schools, compared to 5.0 for non-RTT states (Appendix A, Figure A. 14). 
Therefore, it may not be surprising that the practice in this area with the largest difference 
between the two groups of states fell in this subtopic — implementing or providing technical 
assistance on strategies to recruit and retain effective teachers (such as financial incentives). On 
average, early RTT states used 50 percent of the components of this practice and non-RTT states 
used 13 percent (Figure IV. 17). 

The two practices in this area with the largest differences between later and non-RTT states 
were: (1) providing technical assistance to the lowest-achieving schools or LEAs on improving 
the quality of professional development (with 86 percent of later RTT states and 63 percent of 
non-RTT states using this practice) and (2) having the authority to take over failing schools (with 
57 percent of later RTT states and 34 percent of non-RTT states using this practice) (Figure 
IV. 17). 

We found several practices in this area for which states reported using most of each 
practice’s components: (1) providing training on analyzing data to improve instruction, (2) 
providing training to the lowest-achieving schools or LEAs on identifying and implementing 
effective curricula, instructional strategies, or school intervention models, or developing and 
implementing a school improvement plan, and (3) having administrative structures to support 
turnaround efforts (Figure IV. 17). In contrast, very few states reported having teacher tenure 
rules that affect placement in or removal from the lowest-achieving schools (Figure IV. 17). 
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Figure IV. 17. Use of policies and practices aligned with RTT objectives on 
school turnaround 
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Source: Interviews with state administrators in spring 2013. 

Note: This figure has a separate panel for each subtopic. As described in Appendix A, for each policy or practice 

in the RTT application criteria for which we identified one or more interview questions aligned with the 
policy or practice, we calculated the percentage of interview questions with a “yes” response as a measure 
of the percentage of components each state used. The height of each bar represents the average 
percentage of the components of the policy or practice that each group of states used. A range is provided 
for the sample sizes because nonresponse varied across items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 
practice. 

ELLs = English language learners; LEA = local education agency; PD = professional development; TA = technical 
assistance. 
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F. Early RTT states reported using more charter school practices than non- 

RTT states 

RTT intends to foster an environment in which charter and other innovative schools can 
grow and thrive. One part of Section F of the RTT application, State Reform Conditions, focused 
on ensuring successful conditions for high-performing charter schools and other innovative 
schools. 36 The application focused on two subtopics related to charter schools: (1) eliminating 
restrictions on charter school creation and enrollment, and (2) refining charter school 
authorization and monitoring processes. The state interviews for the evaluation asked about four 
policies and practices aligned with RTT objectives in this area (Table IV. 6). 

Table IV.6. Policies and practices aligned with RTT objectives on charter 
schools, by subtopic 


Eliminating restrictions on charter school creation and enrollment 


Having no restrictions on the creation of new charter schools or charter enrollment 3 


Refining charter school authorization and monitoring processes 


In considering applications for new charter schools, giving priority to schools that propose to address the needs of 
ELLs or that target ELLs 

Monitoring the academic performance of charter schools 3 
Monitoring the non-academic performance of charter schools 3 

Source: Section F of the RTT application (U.S. Department of Education 2010); interviews with state administrators 
in spring 2012 and spring 2013. 

Note: The spring 2012 and 2013 state interviews asked whether these policies and practices were used during 

the current school year (either 201 1-2012 or 2012-2013). The spring 2012 state interview asked whether a 
subset of these policies and practices were used during the 2007-2008 school year. See Appendix C for a 
list of interview questions that were aligned with the RTT policies and practices in this table. See the RTT 
application for a detailed description of each subtopic. 

3 The spring 2012 state interview asked whether this policy or practice was used during the 2007-2008 school year. 
ELLs = English language learners. 

In spring 2013, early RTT states reported using more RTT-promoted policies and 
practices in the charter school area than non-RTT states. Early RTT states reported using an 
average of 2.4 out of 4 policies and practices in this area, compared to 1.6 for non-RTT states 
(Figure IV. 18). For three of these four charter school policies and practices, no significant 
difference between the groups existed before the awarding of RTT grants. (Baseline data were 
not collected for the fourth practice, so we could not determine whether there were baseline 
differences for this topic area as a whole.) 


36 The other reform areas mentioned in Section F of the RTT application (Making Education Funding a Priority and 
Demonstrating Other Significant Reform Conditions) did not focus on charter schools and are excluded from this 
analysis. 
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Figure IV.18. Use of policies and practices aligned with RTT objectives on 
charter schools 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV.6. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 
four examined) that were aligned with the RTT application criteria. The number inside each dot is the 
number of states represented by the dot; dots that represent only one state have no number inside. For 
example, five non-RTT states reported using two of the four charter school policies and practices 
aligned with the RTT application criteria. For three of the policies and practices, a “yes” response 
received one point. In the fourth case, it was possible for a state to receive a fraction of one point. See 
Appendix A for details on how we determined the number of policies and practices for each state. The 
dashed line denotes the average number of policies and practices for each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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In spring 2013, for one of the two charter school subtopics, early RTT states reported 
using more RTT-promoted policies and practices than non-RTT states. Results for the two 

subtopics are shown in Appendix A, Figures A. 15 and A. 16, and results for individual policies 
and practices are shown below in Figure IV. 19. Early RTT states reported using an average of 
1 .9 out of 3 policies and practices in the subtopic focused on refining authorization and 
monitoring processes, compared to 1.3 for non-RTT states (Appendix A, Figure A. 16). 

Therefore, it may not be a surprise that the two practices in this area with the largest differences 
between these groups of states were from this subtopic: (1) monitoring the non-academic 
performance of charter schools (with early RTT states using, on average, 77 percent of the 
components of this practice and non-RTT states using 50 percent), and (2) monitoring the 
academic perfonnance of charter schools (with 100 percent of early RTT states and 74 percent of 
non-RTT states using this practice) (Figure IV. 19). 

The two practices with the largest differences between later RTT and non-RTT states in this 
area were: (1) having no restrictions on the creation of new charter schools or charter enrollment 
(with 57 percent of later RTT states and 38 percent of non-RTT states using this practice), and 
(2) monitoring the non-academic performance of charter schools (with later RTT states using, on 
average, 68 percent of the components of this practice and non-RTT states using 50 percent) 
(Figure IV. 19). 

We found that most states reported monitoring the academic performance of charter schools 
(Figure IV. 19). However, very few states reported giving priority to schools that proposed to 
address the needs of ELLs or that targeted ELLs when considering applications for new charter 
schools (Figure IV. 19). 
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Figure IV.19. Use of policies and practices aligned with RTT objectives on 
charter schools 
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Source: Interviews with state administrators in spring 2013. 

Note: This figure has a separate panel for each subtopic. As described in Appendix A, for each policy or practice 

in the RTT application criteria for which we identified one or more interview questions aligned with the 
policy or practice, we calculated the percentage of interview questions with a “yes” response as a measure 
of the percentage of components each state used. The height of each bar represents the average 
percentage of the components of the policy or practice that each group of states used. A range is provided 
for the sample sizes because nonresponse varied across items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 

practice. 

ELLs = English language learners. 
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G. Summary 

This chapter examined the extent to which RTT and other states reported using RTT- 
promoted policies and practices in spring 2013 in six areas. 

Early RTT states reported using more RTT-promoted policies and practices than non-RTT 
states in four of six areas (the areas of state capacity and data systems were the exceptions), with 
differences ranging from 0.8 to 6.6 policies and practices per area (Figure IV. 20). Later RTT 
states also reported using more RTT-promoted policies and practices than non-RTT states, but 
only in one of the six areas (the area of teacher and principal certification and evaluation), with a 
difference of 5.1 policies and practices (Figure IV.20). These differences were not necessarily 
caused by RTT. In fact, among the three areas for which we collected baseline data on a subset 
of policies and practices, we found that, for one area, a difference between early RTT and non- 
RTT states existed at baseline prior to the RTT program, and for two areas, a difference between 
later RTT and non-RTT states existed prior to the RTT program (Table III. 1). 

Although the findings in this chapter focus on averages for the three groups of states (early 
RTT, later RTT, non-RTT), it is important to keep in mind that there was substantial variation in 
use of RTT-promoted policies and practices within each group. As a result, there was 
considerable overlap across groups in all six areas. In other words, some non-RTT states used 
more RTT-promoted policies and practices than the average number used by RTT states (Figures 
IV. 1, IV. 4, IV. 7, IV. 9, IV. 16, and IV. 18); in the figures, these states are the ones in the third 
column of (green) dots that are above the dashed average lines that are shown for the first and 
second columns of (red and blue) dots. In addition, some RTT states used fewer RTT-promoted 
policies and practices than the average number used in non-RTT states (Figures IV. 1, IV. 4, IV. 7, 
IV. 9, IV. 16, and IV. 18); in the figures, these states are the ones in the first and second columns 
of (red and blue) dots that are below the dashed average line that is shown for the third column 
of (green) dots. 

Across all states, use of RTT-promoted policies and practices was highest in the data 
systems area, because states used the most policies and practices relative to what was promoted 
by RTT for this area. Across all states, use of RTT-promoted policies and practices was lowest in 
the teacher and principal certification and evaluation area, because states used the fewest policies 
and practices relative to what was promoted by RTT for this area. States reported using, on 
average, 76 percent of RTT-promoted policies and practices in the data systems area, and 26 
percent of RTT-promoted policies and practices in the teacher and principal certification and 
evaluation area (Figure IV.20). 
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Figure IV.20. Use of policies and practices promoted by RTT, by topic area 



(10 items) Certification (10 items) (4 items) 

and Evaluation 
(39 items) 

| Early RTT States (N=12)^| Later RTT States (N=7)^|Non-RTT States (N=32) 

Source: Interviews with state administrators in spring 2013. 

Note: The total number of policies and practices (shown in parentheses under each bar) differs by topic area. 

The number of policies and practices that we examined for each area was directly related to the amount of 
emphasis placed on each area in the RTT application criteria. This figure reads as follows (using the first 
bar on the left as an example): early RTT states reported using 7 1 percent of the policies and practices in 
the state capacity area, or 7.1 out of 10 policies and practices examined in that area. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 


Across all policies and practices, the practice with the largest difference between early RTT 
and non-RTT states was supporting the implementation of the CCSS by providing funds for 
additional staff or making new technology investments. On average, early RTT states used 7 1 
percent of the components of this practice and non-RTT states used 16 percent (Figure IV. 6). 
The two practices with the largest differences between later RTT and non-RTT states were: (1) 
requiring evaluations to inform decisions about dismissal of teachers (with 71 percent of later 
RTT states and 22 percent of non-RTT states using this practice) (Figure IV. 12), and (2) 
requiring evaluations to inform decisions about dismissal of principals (with 57 percent of later 
RTT states and 10 percent of non-RTT states using this practice) (Figure IV. 13). 


Nearly all states reported using the following practices (1) having an SLDS, and (2) 
identifying teacher shortage areas (Figures IV. 8 and IV. 10). No states reported (1) using results 
from evaluations of certification programs to provide additional funds for, expand, or promote 
certification programs that were shown to be effective for teachers; (2) doing the same for 
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certification programs that were shown to be effective for principals; and (3) publicly reporting 
results from evaluations of certification program effectiveness for principals (Figure IV. 15). 

H. Comparing and contrasting spring 2013 and spring 2012 findings 

For the most part, the spring 2013 findings presented in this chapter were the same as the 
spring 2012 findings presented in an earlier report from this evaluation (Dragoset et al. 2015). 
There was one key difference between the two sets of findings, which was that the areas for 
which early RTT states reported using more RTT-promoted policies and practices than non-RTT 
states differed between the two years. In particular: 

• In spring 2012, early RTT states reporting using more RTT-promoted policies and practices 
than non-RTT states in five of six areas (the area of school turnaround was the exception). In 
contrast, in spring 2013, early RTT states reporting using more RTT-promoted policies and 
practices than non-RTT states in four of six areas (the areas of state capacity and data 
systems were the exceptions). 

Our data do not allow us to determine the reason for this difference in findings across years. 
However, Chapter VI provides potential explanations for the pattern of findings that we 
observed. 


58 



V. EXTENT TO WHICH STATES FOCUS ON ENGLISH LANGUAGE LEARNERS 
IN THEIR USE OF POLICIES AND PRACTICES PROMOTED BY RACE TO 
THE TOP 


English language learners (ELLs) are of particular interest to this evaluation for two reasons. 
First, they have historically had lower academic achievement than other students. Since 2002, 
ELLs have scored lower than other students on the NAEP reading exam (National Center for 
Education Statistics 2014). Second, the RTT program emphasized prioritizing the academic 
achievement of high-needs students, including ELLs, as a way to address this achievement gap. 

In particular, the RTT application criteria asked states to (1) make data on ELLs available to 
evaluate the effectiveness of instructional materials, strategies, and approaches for ELLs, and (2) 
provide supports and professional development to teachers and principals to ensure that ELLs 
acquire language skills to master academic content (U.S. Department of Education 2010). For 
these reasons, ED’s Office of English Language Acquisition requested that part of this evaluation 
focus on how states used the policies and practices promoted by RTT to address the needs of 
ELLs. 

In this chapter, we assess the extent to which RTT and other states reported focusing on 
ELLs in their use of policies and practices promoted by RTT. We present results from four types 
of analyses: 

1. We compared RTT and other states’ use of ELL-focused policies and practices aligned with 
the RTT application criteria. 

It is possible that states with higher ELL populations and higher achievement gaps between 
ELLs and other students prioritized ELL-focused education refonns more than other states. 
Because of this, comparing RTT and other states’ use of ELL-focused policies and practices 
overall might obscure important differences between RTT and other states that exist only in 
states with higher ELL populations or ELL achievement gaps. For this reason, we conducted the 
following analyses: 

2. We compared RTT and other states’ use of ELL-focused policies and practices within each 
of the following four groups: 

a. States with higher ELL populations (defined as states with percentages of ELLs above 
the median). For example, within the group of states with higher ELL populations, we 
compare early RTT states to non-RTT states; 

b. States with lower ELL populations (defined as states with percentages of ELLs below 
the median); 

c. States with higher ELL achievement gaps (defined as states with achievement gaps 
above the median); and 

d. States with lower ELL achievement gaps (defined as states with achievement gaps below 
the median). 

To examine whether states with higher ELL populations and higher ELL achievement gaps 
prioritized ELL-focused education reforms more than other states, we conducted the following 
analyses: 
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3. Within each of the following groups of states — early RTT, later RTT, and non-RTT — we 
compared use of ELL-focused policies and practices for states that had higher and lower 
ELL populations. For example, within the group of early RTT states, we compared states 
with higher ELL populations to states with lower ELL populations. 

4. Within each of the following groups of states — early RTT, later RTT, and non-RTT — we 
compared use of ELL-focused policies and practices for states that had higher and lower 
ELL achievement gaps. 

Readers interested in specific examples of the individual ELL-focused policies and practices 
included in these analyses may consult Appendix C. 

Table V.l shows descriptive statistics on the ELL population and ELL achievement gap for 
each group of states (early RTT, later RTT, and non-RTT). On average across all states, ELLs 
made up 6.2 percent of the student body and scored 22.4 points lower than other students on the 
NAEP exam. Early, later, and non-RTT states had similar ELL populations on average (6.1 or 
6.2 percent of students), but there was more variation among later RTT states than other states. 
There were no significant differences between early RTT and non-RTT states, or between later 
RTT and non-RTT states, with respect to the average ELL achievement gap. However, later RTT 
states had a higher average ELL achievement gap (28.8) than early RTT states (20.2). As a 
result, most later RTT states (five of seven) were classified as having a higher ELL achievement 

gap- 


Table V.l. Distribution of ELL population and ELL achievement gap 



10th percentile 1.9 2.7 1.3 

50th percentile 5.4 6.5 3.9 

90th percentile 10.9 8.9 14.6 

Mean 6.2 6.1 6.1 

Number of states that had higher and lower ELL populations 


Higher 

26 

7 

3 

16 

Lower 

25 

5 

4 

16 

Distribution of ELL achievement gap 

10th percentile 

13.0 

14.0 

14.0 

10.0 

50th percentile 

24.0 

22.0 

32.5 

23.0 

90th percentile 

32.0 

29.0 

34.0 

30.0 

Mean 

22.4 

20.2 

28.8 

22.0 


1.8 

5.2 
11.0 

6.2 


Number of states that had higher and lower ELL achievement gaps 

Higher 25 a 6 5 14 

Lower 21 6 1 14 

Number of states 45-51 12 6-7 27-32 


Source: Common Core of Data (CCD); National Assessment of Educational Progress (NAEP) scores. 

Note: We calculated the ELL population using CCD data on the percentage of public school students participating 

in programs for ELLs in 2007-2008. We calculated ELL achievement gaps as the average score for non- 
ELLs minus the average score for ELLs on the 2007 NAEP 4th grade math exam. As a point of reference, a 
20-point difference on this exam is equivalent to the difference between a student who scored at the 25th 
percentile nationally (a score of 222) and one who scored at the 50th percentile (a score of 242). A 20-point 
difference on this exam is also equivalent to an effect size of 0.7 (the standard deviation was 29 points). 
States were classified into higher and lower groups based on whether their value (for either the ELL 
population or the ELL achievement gap) was above or below the median value across all states. See 
Chapter II for more details on how we classified states into groups. A range is provided for the sample size 
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because missing data varied across items. There were no statistically significant differences in 2007-2008 
between early RTT and non-RTT states, or between later RTT and non-RTT states. 
a The number of states with a higher achievement gap is not equal to the number of states with a lower gap because 
five states had the median value (and were assigned to the above-median group). 

ELL = English language learner. 

The evaluation’s state interviews asked about 12 ELL-focused policies and practices aligned 
with RTT objectives (Table V.2). 

Table V.2. ELL-focused policies and practices aligned with RTT objectives 

Providing districts with funding, materials, training, technical assistance, or other supports to aid in the use of ELL- 
related data 

Having teacher assignment laws or policies that include financial incentives to recruit and retain teachers with ELL 
expertise 

Training the lowest-achieving schools or local education agencies in identifying and implementing strategies to 
address the need of ELLs 

Having state-level staff or consultants to support turnaround schools and districts in working with ELLs 

In considering applications for new charter schools, prioritizing schools that propose to address the needs of ELLs 
or that target ELLs 

Monitoring charter school performance based on the student populations (such as ELLs) served 
Prioritizing the adoption and implementation of supports to ELLs 

Providing targeted support to ELLs or working with intermediaries to provide support to ELLs 

Implementing organizational or administrative changes to improve capacity to support ELLs 

Supporting districts or schools in implementing the Common Core State Standards with ELLs 

Ensuring state longitudinal data system contains program participation information about ELLs 

Using state longitudinal data system to inform and engage stakeholders and support decision makers in 
continuous improvement efforts for ELLs 

Source: RTT application (U.S. Department of Education 2010); interviews with state administrators in spring 2012 
and spring 2013. 

Note: See Appendix C for a list of the interview questions that were aligned with the ELL-focused policies and 

practices in this table. All the policies and practices listed in this table were included in the main analyses 
described in Chapter IV, but some of them are not listed in the Chapter IV tables because they were 
included in a broader policy or practice that is listed in those tables. 

ELL = English language learner. 

In spring 2013, early RTT states reported using more ELL-focused policies and 
practices promoted by RTT than non-RTT states. Early RTT states reported using an average 
of 7.0 of 12 ELL-focused policies and practices, compared to 5.4 for non-RTT states (Figure 
V. 1). The ELL-focused practice with the largest difference between early RTT and non-RTT 
states was providing districts with funding, materials, training, technical assistance, or other 
supports to aid in the use of ELL-related data. On average, 91 percent of early RTT states used 
this practice, compared with 45 percent of non-RTT states (Figure V.2) (Figure V.3 shows 
similar infonnation for other ELL-focused practices.) 
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Figure V.l. Use of ELL-focused policies and practices aligned with RTT 
objectives 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table V.2. Each dot in this figure 

represents the number of states that reported using a particular number of ELL-focused policies and 
practices (out of 12 examined) that were aligned with the RTT application criteria. The number inside each 
dot is the number of states represented by the dot; dots that represent only one state have no number 
inside. For example, two non-RTT states reported using 6 of 12 ELL-focused policies and practices aligned 
with the RTT application criteria. For 6 of the policies and practices, a “yes” response received one point. In 
the other 6 cases, it was possible for a state to receive a fraction of one point. See Appendix A for details 
on how we determined the number of ELL-focused policies and practices for each state. The dashed line 
denotes the average number of ELL-focused policies and practices for each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 

The ELL-focused practice with the largest difference between later RTT and non-RTT states 
was monitoring charter school perfonnance based on the student populations (such as ELLs) 
served. On average, 71 percent of later RTT states and 29 percent of non-RTT states used this 
practice (Eigure V.2). 

States reported using most components of two ELL-focused practices: (1) providing targeted 
support to ELLs or working with intermediaries to provide support to ELLs, and (2) training the 
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lowest-achieving schools or local education agencies in identifying and implementing strategies 
to address the need of ELLs (Figures V.2 and V.3). 

In contrast, states reported using very few components of the following ELL-focused 
practices: (1) having teacher assignment laws or policies that include financial incentives to 
recruit and retain teachers with ELL expertise and (2) prioritizing charter school applications that 
proposed to address the needs of ELLs (Figure V.2). 


Figure V.2. Use of the first six ELL-focused policies and practices aligned 
with RTT objectives 
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Policy or Practice Aligned with RTT Objectives 


I Early RTT States (N=1 1-12) | Later RTT States (N=6-7) | Non-RTT States (N=31-32) 


Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each ELL-focused policy or practice in the RTT application criteria for 

which we identified one or more interview questions aligned with the policy or practice, we calculated the 
percentage of interview questions with a “yes” response as a measure of the percentage of components 
each state used. The height of each bar represents the average percentage of the components of the policy 
or practice that each group of states used. A range is provided for the sample sizes because nonresponse 
varied across items. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or 

practice. 

ELL = English language learner. 
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Figure V.3. Use of the last six ELL-focused policies and practices aligned 
with RTT objectives 



Improve Capacity to about ELLs A Decisions' 1 

Support ELLs A 

Policy or Practice Aligned with RTT Objectives 


| Early RTT States (N=1 2) | Later RTT States (N=7) ^ Non- RTT States (N=32) 

Source: Interviews with state administrators in spring 2013. 

Note: As described in Appendix A, for each ELL-focused policy or practice in the RTT application criteria for which we 

identified one or more interview questions aligned with the policy or practice, we calculated the percentage of 
interview questions with a “yes” response as a measure of the percentage of components each state used. The 
height of each bar represents the average percentage of the components of the policy or practice that each 
group of states used. 

A Multiple interview questions were used to assess whether states used all of the components of this policy or practice. 

ELL = English language learner; CCSS = Common Core State Standards; SLDS = State longitudinal data system. 

Among states with lower ELL populations, later RTT states reported using more ELL- 
focused policies and practices promoted by RTT than non-RTT states. Later RTT states with 
lower ELL populations reported using an average of 6.9 of 12 ELL-focused policies and 
practices, compared to 4.9 for non-RTT states with lower ELL populations (Figure V.4). 


Among states with lower ELL achievement gaps, early RTT states reported using more 
ELL-focused policies and practices promoted by RTT than non-RTT states. Early RTT 
states with lower ELL achievement gaps reported using an average of 6.9 of 12 ELL-focused 
policies and practices, compared to 5.4 for non-RTT states with lower ELL achievement gaps 
(Figure V.5). 
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Within each group of states — early RTT, later RTT, and non-RTT — there were no 
differences in use of ELL-focused policies and practices promoted by RTT between states 
with higher and lower ELL populations or between states with higher and lower ELL 
achievement gaps. Early RTT, later RTT, and non-RTT states with higher ELL populations 
reported using 7.5, 5.6, and 5.9 of 12 ELL-focused policies and practices, while those states with 
lower ELL populations reported using 6.2, 6.9, and 4.9 policies and practices (Figure V.4). Early 
RTT, later RTT, and non-RTT states with higher ELL achievement gaps reported using 7.0, 6.0, 
and 5.2 of 12 ELL-focused policies and practices, while those states with lower ELL 
achievement gaps reported using 6.9, 8.7, and 5.4 policies and practices (Figure V.5). 


Figure V.4. Use of ELL-focused policies and practices aligned with RTT 
objectives, by ELL population 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table V.2. Each column in the figure 

shows the number of ELL-focused policies and practices that states in each group (early RTT, later RTT, 
non-RTT) reported using, by states that had higher and lower ELL populations. Each dot in this figure 
represents the number of states that reported using a particular number of ELL-focused policies and 
practices (out of 12 examined) that were aligned with the RTT application criteria. The number inside each 
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dot is the number of states represented by the dot; dots that represent only one state have no number 
inside. For example, two non-RTT states that had lower ELL populations reported using 6 of 12 ELL- 
focused policies and practices aligned with the RTT application criteria. For 6 of the policies and practices, 
a “yes” response received one point. In the other 6 cases, it was possible for a state to receive a fraction of 
one point. See Appendix A for details on how we determined the number of ELL-focused policies and 
practices for each state. The dashed line denotes the average number of ELL-focused policies and 
practices for each group of states. There were no statistically significant differences at the 0.05 level using 
a two-tailed test between higher and lower ELL population states with the same RTT status. Among states 
with higher ELL populations, there were no statistically significant differences at the 0.05 level using a two- 
tailed test between RTT and non-RTT states. 

‘Significantly different from non-RTT states with lower ELL populations at the 0.05 level, two-tailed test. 

Figure V.5. Use of ELL-focused policies and practices aligned with RTT 
objectives, by ELL achievement gap 
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Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table V.2. Each column in the figure 

shows the number of ELL-focused policies and practices that states in each group (early RTT, later RTT, 
non-RTT) reported using, by states that had higher and lower achievement gaps between ELLs and other 
students. Each dot in this figure represents the number of states that reported using a particular number of 
ELL-focused policies and practices (out of 12 examined) that were aligned with the RTT application criteria 
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The number inside each dot is the number of states represented by the dot; dots that represent only one 
state have no number inside. For example, two non-RTT states that had a higher ELL achievement gap 
reported using about 4 of 12 ELL-focused policies and practices aligned with the RTT application criteria. 
For 6 of the ELL-focused policies and practices, a “yes” response received one point. In the other 6 cases, 
it was possible for a state to receive a fraction of one point. See Appendix A for details on how we 
determined the number of ELL-focused policies and practices for each state. The dashed line denotes the 
average number of ELL-focused policies and practices for each group of states. There were no statistically 
significant differences at the 0.05 level using a two-tailed test between higher and lower ELL achievement 
gap states with the same RTT status. Among states with higher ELL achievement gaps, there were no 
statistically significant differences at the 0.05 level using a two-tailed test between RTT and non-RTT 
states. 

‘Significantly different from non-RTT states with lower ELL achievement gaps at the 0.05 level, two-tailed test. 

Thus, it does not appear to be the case that states with higher ELL populations and higher 
achievement gaps between ELLs and other students prioritized ELL-focused education reforms 
more than other states. However, there is some evidence that RTT states used ELL-focused 
policies and practices more than other states. 

For the most part, the spring 2013 findings presented in this chapter were similar to 
the spring 2012 findings presented in an earlier report from this evaluation (Dragoset et al. 
2015). In a few cases, the spring 2013 findings were statistically significant while the spring 
2012 findings were not, but in both years, the differences between groups of states with respect 
to the number of ELL-focused practices used were small (one to two practices). 
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Despite the substantial resources devoted to RTT and its importance as an initiative seeking 
to spur nationwide changes to educational policies and practices, no studies have examined the 
relationship between the RTT program and student outcomes. In addition, few studies have 
examined (a) the implementation of RTT-promoted policies and practices in all of the topic areas 
described in the RTT application and (b) whether the policies and practices used by RTT states 
differ from those used by other states. This evaluation sought to address these gaps in the 
existing literature by attempting to examine the relationship between RTT grant receipt and 
student outcomes and examining the extent to which RTT and other states reported using RTT- 
promoted policies and practices in all of the topic areas described in the RTT application. We 
also assessed the extent to which RTT and other states reported focusing on English language 
learners (ELLs) in their use of policies and practices promoted by RTT. We focused on policies 
and practices that states reported using in spring 2013, and also examined changes in use 
between 2011-2012 and 2012-2013. For certain policies and practices, we also examined 
changes between 2007-2008 (before the RTT program was announced) and 2012-2013. 

In this chapter, we summarize our findings on RTT implementation and on the relationship 
between RTT grant receipt and student outcomes. We then lay out several questions of possible 
interest and potential explanations for these findings. 

A. RTT implementation findings 

In spring 2013, early RTT states reported using more policies and practices promoted by 
RTT, on average, than non-RTT states in four of the six areas considered (state capacity and data 
systems were the exceptions). This finding is consistent with previous studies of RTT 
implementation that suggested that RTT states were more likely to use RTT-promoted policies 
and practices than other states (Dragoset et al. 2015; Government Accountability Office 2011; 
National Council on Teacher Quality 2011; Howell 2015). The difference between early RTT 
and non-RTT states in the average number of RTT-promoted policies and practices used ranged 
from 0.8 to 6.6 across the four areas. However, some of these differences predated award of the 
RTT grants. In particular, early RTT states reported having already used more of some teacher 
and principal certification and evaluation policies and practices promoted by RTT than non-RTT 
states as of 2007-2008, before receiving their RTT grants. For several areas — state capacity, 
standards and assessments, data systems — we did not have infonnation on states’ use of RTT- 
promoted policies and practices prior to the awarding of RTT grants; therefore, for those areas, 
we could not detennine whether some of these differences predated receipt of the RTT grants. 

In spring 2013, later RTT states reported using more policies and practices promoted by 
RTT, on average, than non-RTT states in only one of the six reform areas (teacher and principal 
certification and evaluation), with a difference of 5.1 policies and practices. As with the early 
RTT states, there was some evidence of differences for the teacher and principal certification and 
evaluation area between later RTT and non-RTT states as of 2007-2008, before RTT grants were 
awarded. 

Although the findings reported above focus on averages for each group of states (early RTT, 
later RTT, non-RTT), there was substantial variation in the reported number of RTT-promoted 
practices used within each group. 
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In spring 2013, across all states, use of RTT-promoted practices was highest in the data 
systems area and lowest in the teacher and principal certification and evaluation area. States 
reported using, on average, 76 percent of RTT-promoted practices in the data systems area and 
26 percent of RTT-promoted practices in the teacher and principal certification and evaluation 
area. Nearly all states reported: (1) having an SLDS, and (2) identifying teacher shortage areas. 

In contrast, no states reported: (1) using results from evaluations of certification programs to 
provide additional funds for, expand, or promote certification programs that were shown to be 
effective for teachers; (2) doing the same for certification programs that were shown to be 
effective for principals; and (3) publicly reporting results from evaluations of certification 
program effectiveness for principals. 

In spring 2013, the practice with the largest difference between early RTT and non-RTT 
states was supporting the implementation of the CCSS by providing funds for additional staff or 
making new technology investments. The two practices with the largest differences between later 
RTT and non-RTT states were: (1) requiring evaluations to inform decisions about dismissal of 
teachers, and (2) requiring evaluations to inform decisions about dismissal of principals. 

When we focused on changes over time in states’ use of policies and practices, we found no 
significant differences between RTT and other states (see Appendix D). 

In spring 2013, early RTT states reported using more ELL-focused policies and practices 
promoted by RTT than did non-RTT states. There was no difference between later RTT and non- 
RTT states in the number of ELL-focused policies and practices used. 

Many of the spring 2013 findings presented in this report were the same as the spring 2012 
findings presented in an earlier report from this evaluation (Dragoset et al. 2015). Two key 
differences were: 

1. In spring 2012, early RTT states reported using more RTT-promoted policies and practices 
than non-RTT states in five of six areas (the area of school turnaround was the exception). 

In contrast, in spring 2013, we found differences in four of six areas (the areas of state 
capacity and data systems were the exceptions). 

2. In spring 2012, we found no significant difference in use of ELL-focused policies and 
practices between RTT and other states, whereas in spring 2013, we found that early RTT 
states used more ELL-focused policies and practices than non-RTT states. 

B. Findings on the relationship between RTT grant receipt and student 
outcomes 

The effect of RTT on student outcomes was not clear because trends in outcomes could be 
interpreted as providing evidence of a positive, a negative, or no effect of RTT (see Appendix E). 
Conclusions about the relationship between RTT grant receipt and outcomes depend on 
assumptions made about the pattern of outcomes in the pre-RTT period. In addition, other 
changes happening at the same time as RTT reforms might have affected outcomes. 
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C. Questions of interest and potential explanations for findings 

Readers may have questions about potential explanations for the observed pattern of 
findings. Below, we lay out questions of possible interest and potential explanations for these 
findings. 

Why did early RTT states report using more RTT-promoted policies and practices 
than non-RTT states in spring 2013 in four of six areas examined? We focus here on two 
potential explanations. First, this finding could reflect the effect of the RTT grants on state 
policies and practices. RTT grants aimed to increase use of the policies and practices examined, 
so this pattern of findings could reflect progress toward that goal. Second, this finding could 
reflect differences in use of these policies and practices between the groups of states prior to the 
awarding of the RTT grants. For the three areas for which we could examine the prevalence of 
RTT-promoted policies and practices both before and after RTT grants were awarded, evidence 
suggests that some differences existed prior to the RTT awards. Further, the RTT application 
criteria gave explicit priority to states that had already used policies consistent with the RTT 
goals: “Over half the points that reviewers may award to states are based on states’ 
accomplishments prior to applying” (U.S. Department of Education 2010). Therefore, it is 
possible that the higher use of RTT-promoted practices observed among early RTT states is at 
least partly an artifact of the selection process rather than a result of the RTT grant awards, but 
this does not mean that RTT had no effect on state policies and practices. Rather, RTT may have 
affected policies and practices in many states simply by holding the competition; in other words, 
many states (including both those that eventually received RTT awards and those that did not) 
may have increased their use of RTT-promoted policies and practices after 2007-2008 to better 
position themselves to receive an RTT award. Our finding that use of RTT-promoted policies 
and practices increased between 2007-2008 and 2011-2012 for all states (not just RTT states) in 
three areas is consistent with this hypothesis. 

Why did early RTT states not report using more RTT-promoted policies and practices 
than non-RTT states in the state capacity and data systems areas in spring 2013? We focus 
here on two potential explanations for this finding. First, it is possible that RTT did not have an 
effect on states’ practices in these areas. Second, it is possible that particular state capacity 
policies and practices (such as creating a new office or department to support the implementation 
of state education refonn priorities) were one-time events that occurred in 2011-2012 and thus, 
did not need to occur again in 2012-2013. As mentioned in Chapter II, the results from spring 
2013 represent only the policies and practices used during the specific time period between 
spring 2012 and spring 2013, not the full set of policies and practices used at any point after 
receipt of an RTT grant. Our finding that use of state capacity policies and practices by early and 
later RTT states decreased between 2011-2012 and 2012-2013 is consistent with this 
hypothesis. Third, it is possible that use of data systems practices are on the rise more broadly 
(even in states that did not receive RTT grants), so that we did not observe the RTT program 
resulting in higher levels of use among RTT states relative to non-RTT states. Data systems 
practices may be rising nationally, in part, due to the Statewide Longitudinal Data Systems 
(SLDS) grant program which supports development, implementation, and expansion of K-12 
longitudinal data systems. Non-RTT states reported using more policies and practices in the data 
systems area in 2013 than in 2012, suggesting greater usage of data systems practices even in 
states that did not receive RTT grants. 
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Why did later RTT states not report using more RTT-promoted policies and practices 
than non-RTT states in five of the six areas examined in spring 2013? There are several 
potential explanations for this finding. First, later RTT grants were smaller and more limited in 
focus. Later RTT states generally focused on a subset of the six areas addressed by early RTT 
states. All seven later RTT states focused on the areas of state capacity and standards and 
assessments, six focused on data systems and teacher and principal certification and evaluation, 
two focused on charter schools, and one focused on school turnaround. This more limited focus 
might make it less likely for differences between later RTT states and non-RTT states to exist for 
any given area. However, our finding that later RTT states reported using more RTT-promoted 
teacher and principal certification and evaluation practices than did non-RTT states is consistent 
with the fact that six of the seven later RTT states focused on this area. 

Second, since the later RTT states did not receive grants in the first or second rounds of 
competition (despite having applied in earlier rounds), our findings may reflect limitations in 
these states’ capacity for policy changes. 

Finally, our statistical power (that is, our ability to detect differences between the groups of 
states) was lower for the comparisons of later RTT and non-RTT states than for the comparisons 
of early RTT and non-RTT states, due in part to the smaller sample size. Therefore, even if there 
were differences between later RTT and non-RTT states of a similar size as the significant 
differences we found between early RTT and non-RTT states, they might not have been 
statistically significant. However, in five of six areas, the differences between later RTT and 
non-RTT states were smaller than the differences between early RTT and non-RTT states. 

Why did the areas for which early RTT states reported using more RTT-promoted 
policies and practices than non-RTT states in spring 2013 differ from those reported in 
spring 2012? We focus here on two potential explanations. First, it is possible that RTT states 
may have intentionally focused on certain policy and practice areas in some years and 
emphasized other areas in subsequent years. Our finding that early RTT states used more school 
turnaround policies and practices than non-RTT states in spring 2013 but not spring 2012, and 
our finding that early RTT states used more state capacity and data systems policies and 
practices than non-RTT states in spring 2012 but not spring 2013, are consistent with this 
hypothesis. Second, it is possible that particular policies and practices (such as creating a new 
office or department to support the implementation of state education reform priorities) were 
one-time events that RTT states used in 2011-2012 and thus, were not needed again in 2012- 
2013. Our finding that use of state capacity policies and practices by early and later RTT states 
decreased between 2011-2012 and 2012-2013 is consistent with this hypothesis. 

Although we cannot definitively accept or reject any of these possible explanations for these 
findings, we offer them as starting points for future investigations into the implementation of 
policies and practices that RTT promoted. 

Why isn’t it possible to provide credible estimates of the effect of RTT on student 
outcomes? Recognizing that RTT represents a significant investment in education reform, we 
carefully considered whether it was possible to credibly estimate the effect of RTT on student 
outcomes. We concluded that it was not, for the following reasons: (1) conclusions about the 
effect of RTT differed depending on what assumptions we made about the pattern of outcomes in 
the pre-RTT period, and it was difficult to detennine the true pattern of outcomes during that 
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period because NAEP data were available for all states for only four years in that period; and (2) 
even if it were possible to determine the true pattern of outcomes during the pre-RTT period, we 
could not conclude that receipt of an RTT grant caused changes in student outcomes because 
other changes taking place at the same time as RTT reforms might also have affected student 
outcomes. 
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In Chapter IV, we summarized the extent to which states reported using the policies and 
practices promoted by RTT. In Section A of this appendix, we describe how we analyzed the 
state interview data. In Sections B through G, we present additional figures that are directly 
related to the analyses presented in Chapter IV. We focus on the same six topic areas addressed 
in Chapter IV. For each area, we present a series of figures, one for each subtopic, showing 
states’ use of the policies and practices aligned with the RTT application criteria for that 
subtopic, similar to the figures shown in Chapter IV for each topic area. 

A. Analysis methods 

Given the large number of questions in the interviews, it is difficult to discern broad patterns 
or fonn overall conclusions by separately examining responses to individual questions. 
Therefore, we analyzed data from the interviews using methods (described below) designed to 
provide infonnation about broad patterns in the data. Readers interested in the responses to 
specific interview questions can refer to Appendix B. 

The process of summarizing findings involved several steps: 

1 . Selecting subtopics. For each of the six topic areas in the RTT application, we selected 
subtopics of interest using the RTT application criteria as a guide. For example, the 
standards and assessments section of the application criteria included subsections on (1) 
developing and adopting common standards; (2) developing and implementing common, 
high-quality assessments; and (3) supporting the transition to enhanced standards and high- 
quality assessments. We used each of those three subsections as subtopics in our analysis. 

2. Selecting questions aligned with the RTT application criteria. For each subtopic, we 
used a systematic approach to select interview questions that aligned with the policies and 
practices that RTT sought to affect in that area. First, two Mathematica researchers 
independently selected questions corresponding to each policy or practice based on whether 
he or she determined them to be aligned with the RTT application (the agreement rate was 
97 percent). 37 A third Mathematica researcher resolved any differences of opinion. We 
determined the topic area and subtopic into which each interview question fell based on the 
section of the RTT application with which it aligned. By design, the modules in the state 
interviews directly map to sections of the application. We did not use an interview question 
for more than one subtopic because doing so would have resulted in the question being 
overweighted in the overall topic area. When a question could be used for more than one 
subtopic, we assigned it to the subtopic (and corresponding section of the application) with 
which it was best aligned. The interview questions addressed all six topic areas and all but 2 
of 18 subtopics from the application criteria. 

3. Constructing policy and practice variables from interview questions. For each policy or 
practice in the RTT application criteria for which we identified one or more relevant 


37 It was important to have two researchers independently select questions to ensure that the questions selected for 
each subtopic were aligned with the RTT application criteria. Measurement of the extent to which the first and 
second researchers agreed on which questions were aligned with the RTT application criteria is called “inter-rater 
reliability” in statistics. Inter-rater reliability is traditionally measured using the percentage agreement rate, 
calculated as the number of questions for which the first and second researchers agreed on whether the question was 
aligned with the RTT application criteria, divided by the total number of interview questions (Gwet 2014). 
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questions, we calculated the percentage of questions to which the state responded “yes.” 

This variable measures the percentage of components of the policy or practice that each 
state used. For instance, states that responded “yes” to all of the questions aligned with the 
policy or practice used 100 percent of the components, states that responded “yes” to none 
of the questions used 0 percent of the components, and states that responded “yes” to some 
of the questions used between 0 and 100 percent. 38 This process resulted in 10 policies and 
practices for state capacity, 10 for standards and assessments, 8 for data systems, 39 for 
teacher and principal certification and evaluation, 10 for school turnaround, and 4 for charter 
schools. The emphasis (that is, the number of policies and practices) given to each area and 
the specificity of the policies and practices within each area directly relates to how much 
emphasis and specificity was provided in the RTT application criteria for these areas. 39 

4. Summing the policies and practices for each state. To detennine each state’s progress in 
using policies and practices aligned with RTT, we summed the variables created in step 3. 
This sum was calculated separately for each subtopic. We then summed across subtopics to 
create a sum for each topic area. 40 If a particular state was missing values for a particular 
policy or practice, we took the mean of the non-missing policies and practices and 
multiplied it by the total number of policies and practices for the overall area. For example, 
for the state capacity area, which has 10 policies and practices, if a state had data available 
for 5 policies and practices, and reported using 2 of them, the number of the state’s reported 
used policies and practices would be equal to (2/5)* 10. Across all states and all subtopics, 
the average percentage of policies and practices that were missing was low, just 2.4 
percent. 41 


38 Many questions were originally structured with two response options: “yes” (recoded to a value of 100 percent) 
indicating that the state reported using the policy or practice, and "no” (recoded to a value of 0 percent) indicating 
that the state did not report using the policy or practice. In some cases, however, it was necessary to combine 
multiple questions to determine whether a state reported fully using a particular policy or practice. For example, one 
criterion in the application is that teachers be evaluated annually, but the interview asked separate questions about 
the frequency of evaluation for several types of teachers. In this case, we required that a state respond “yes” to all 
these questions to receive a value of 100 for the variable corresponding to this policy or practice. In other cases, any 
one of several questions could be counted as partial evidence of a state’s use of a policy or practice aligned with the 
RTT application criteria. For example, the application criteria specified that the state use support from a broad group 
of stakeholders to implement its education policy plans. The interview included eight separate questions about 
whether each of eight types of stakeholders (such as local school boards and community organizations) were 
involved in defining the state’s education policy plans. In these cases, a state received less than 100 percent (in this 
example, 12.5 percent, or one-eighth of 100 percent) for each “yes” response. This approach helped to ensure that 
we did not overweight some interview questions relative to how they were represented in the application criteria. 

39 For example, the teacher and principal certification and evaluation area was more heavily emphasized in the RTT 
application criteria than any other area. This emphasis is demonstrated by the number of pages devoted to this area 
in the RTT application (11.5 pages, versus 8, 3.5, 3, 2.5, and 2 pages for state capacity, standards and assessments, 
data systems, school turnaround, and charter schools), the percentage of points received by this section of the 
application (28 percent, versus 15, 14, 9, 10, and 8 percent for the other areas), and the number of criteria in this 
section of the application (9 criteria, versus 7, 4, 3, 3, and 1 for the other areas). Correspondingly, our state interview 
was designed to contain more questions aligned with the teacher and principal certification and evaluation section of 
the application than with the other sections. 

40 Thus, one or more interview questions were used to create a variable for each policy or practice, one or more 
policies or practices formed a subtopic, and one or more subtopics formed a topic area (or area, for short). 

41 To assess how our coding of missing data might have affected our results, we conducted a bounding exercise in 
which we re-calculated the results twice: once setting all missing responses to “no” (that is, assuming all missing 


A-3 



APPENDIX A 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


5. Averaging the number of policies and practices across states. For each group of states 
(early RTT, later RTT, non-RTT), we averaged the numbers calculated in step 4. We 
calculated this average number of policies and practices for the three groups of states 
separately for each topic area and subtopic. 

6. Testing differences between groups of states. We conducted statistical tests to assess 
whether the average number of policies and practices reported differed between the early 
RTT and non-RTT states. 42 We also tested whether the average number differed between 
the later RTT and non-RTT states. Because the goal of this analysis was to provide 
descriptive infonnation about the actual levels of policies and practices used by RTT and 
other states in spring 2012 and 2013, the results are reported as raw (that is, unadjusted) 
means; they are not regression-adjusted to account for any pre-existing differences between 
RTT and other states. When reporting findings from this analysis, we focused on the 
statistical significance of differences between RTT and other states (rather than the 
magnitude of differences) to ensure that consistent, objective, and transparent criteria were 
used for reporting findings. One caveat with this approach is that some statistically 
significant differences might not be substantively important; we indicated in the report 
where this might be the case. 

This method of summarizing findings is one way to analyze broad patterns observed in the 
data, and compare use of policies and practices across different groups of states. If variables had 
been constructed differently (for example, if multiple questions that addressed the same policy or 
practice had not been combined into a single variable, but had each been included in the analysis 
as separate variables), the results might change. Therefore, it is important to keep these methods 
in mind when interpreting the results, along with the caveats mentioned above about the 
interview instrument and the wording of the RTT application criteria. 


responses indicated that the policy or practice was not used) and once setting all missing responses to “yes” (that is, 
assuming all missing responses indicated that the policy or practice was used). The results were largely unchanged. 
The magnitude of differences between RTT and other states with respect to the number of RTT-promoted policies 
and practices used were very similar to the magnitudes reported in Chapter IV. In addition, across the 24 statistical 
significance tests conducted as part of this bounding exercise (two for each of the six topic areas when comparing 
early RTT to non-RTT states, and two for each of the six topic areas when comparing later RTT to non-RTT states), 
no results differed from what is shown in Chapter IV. 

42 Although we have the entire population of states in our analysis (as opposed to a sample of states), hypothesis 
testing is still a useful way to account for the possibility that differences in outcomes between the two groups are 
due to randomness rather than due to a “true,” systematic difference. Although there is no source of randomness 
inherent in our study design (such as random sampling or random assignment), there may be other reasons for 
differences in outcomes between the two groups that are functionally random (that is, orthogonal to RTT status and 
other observable covariates). For this analysis, we used a permutation test, which is the non-parametric counterpart 
to a t-test. The statistical power of this test differed by topic area and subtopic because it depended on several 
factors, including the number of interview questions aligned with the RTT application criteria, the number of 
variables constructed from those questions, and the degree to which the variables were correlated with each other. 
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B. State capacity 


Figure A.1. Use of policies and practices aligned with RTT, articulating 
state’s reform agenda subtopic 



(N=12) (N=7) (N=32) 


Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 1. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of five 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For two of the 
policies and practices, a “yes” response received one point. In the other three cases, it was possible fora 
state to receive a fraction of one point. The dashed line denotes the average number of policies and 
practices for each group of states. There were no statistically significant differences between RTT and non- 
RTT states at the 0.05 level using a two-tailed test. 
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Figure A.2. Use of policies and practices aligned with RTT, building strong 
statewide capacity subtopic 



(N=12) (N=7) (N=32) 

Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 1. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of five 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For one policy or 
practice, a “yes” response received one point. In the other four cases, it was possible for a state to receive 
a fraction of one point. The dashed line denotes the average number of policies and practices for each 
group of states. There were no statistically significant differences between RTT and non-RTT states at the 
0.05 level using a two-tailed test. 
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C. Standards and assessments 


Figure A.3. Use of policies and practices aligned with RTT, developing and 
adopting common standards subtopic 



(N=12) (N=7) (N=31) 


Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 2. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of one 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For this policy or 
practice, a “yes” response received one point. The dashed line denotes the average number of policies and 
practices for each group of states. There were no statistically significant differences between RTT and non- 
RTT states at the 0.05 level using a two-tailed test. 
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Figure A.4. Use of policies and practices aligned with RTT, developing and 
implementing common, high-quality assessments subtopic 



(N=12) (N=7) (N=32) 

Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV.2. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of two 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For these policies 
and practices, a “yes” response received one point. The dashed line denotes the average number of 
policies and practices for each group of states. There were no statistically significant differences between 
RTT and non-RTT states at the 0.05 level using a two-tailed test. 
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Figure A.5. Use of policies and practices aligned with RTT, supporting the 
transition to enhanced standards and high-quality assessments subtopic 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 2. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of seven 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For three of the 
policies and practices, a “yes” response received one point. In the other four cases, it was possible for a 
state to receive a fraction of one point. The dashed line denotes the average number of policies and 
practices for each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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D. Data systems 


Figure A.6. Use of policies and practices aligned with RTT, fully 
implementing an SLDS subtopic 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV.3. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of four 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For these policies 
and practices, a “yes” response received one point. The dashed line denotes the average number of 
policies and practices for each group of states. There were no statistically significant differences between 
RTT and non-RTT states at the 0.05 level using a two-tailed test. 

SLDS = state longitudinal data system. 
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Figure A.7. Use of policies and practices aligned with RTT, accessing state 
data and using it to inform key stakeholders subtopic 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV.3. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of two 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For these policies 
and practices, a “yes” response received a fraction of one point. The dashed line denotes the average 
number of policies and practices for each group of states. There were no statistically significant differences 
between RTT and non-RTT states at the 0.05 level using a two-tailed test. 
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Figure A.8. Use of policies and practices aligned with RTT, using data to 
improve instruction subtopic 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV.3. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of two 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For these policies 
and practices, a “yes” response received one point. The dashed line denotes the average number of 
policies and practices for each group of states. There were no statistically significant differences between 
RTT and non-RTT states at the 0.05 level using a two-tailed test. 


A-12 


APPENDIX A 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


E. Teacher and principal certification and evaluation 

Figure A.9. Use of policies and practices aligned with RTT, providing 
high-quality pathways to certification for aspiring teachers and principals 
subtopic 
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Early RTT States 
(N=12) 


Later RTT States 
(N-7) 


Non- RTT States 
(N=32) 


Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 4. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of 14 examined) 
that were aligned with the RTT application criteria. The number inside each dot is the number of states 
represented by the dot; dots that represent only one state have no number inside. For 6 of the policies and 
practices, a “yes” response received one point. In the other 8 cases, it was possible for a state to receive a 
fraction of one point. The dashed line denotes the average number of policies and practices for each group 
of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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Figure A.10. Use of policies and practices aligned with RTT, improving 
teacher and principal effectiveness based on performance subtopic 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 4. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of 1 5 examined) 
that were aligned with the RTT application criteria. The number inside each dot is the number of states 
represented by the dot; dots that represent only one state have no number inside. For 1 1 of the policies and 
practices, a “yes” response received one point. In the other 4 cases, it was possible for a state to receive a 
fraction of one point. The dashed line denotes the average number of policies and practices for each group 
of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 


A-14 


APPENDIX A 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure A.11. Use of policies and practices aligned with RTT, ensuring 
equitable distribution of effective teachers and principals subtopic 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 4. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of four 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For two of the 
policies and practices, a “yes” response received one point. In the other two cases, it was possible for a 
state to receive a fraction of one point. The dashed line denotes the average number of policies and 
practices for each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 


A-15 


APPENDIX A 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure A.12. Use of policies and practices aligned with RTT, improving the 
effectiveness of teacher and principal preparation programs subtopic 

o 



Note: The policies and practices summarized in this figure are presented in Table IV.4. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of six examined) 
that were aligned with the RTT application criteria. The number inside each dot is the number of states 
represented by the dot; dots that represent only one state have no number inside. For these policies and 
practices, a “yes” response received one point. The dashed line denotes the average number of policies 
and practices for each group of states. There were no statistically significant differences between RTT and 
non-RTT states at the 0.05 level using a two-tailed test. 
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F. School turnaround 


Figure A.13. Use of policies and practices aligned with RTT, intervening in 
the lowest-achieving schools and LEAs subtopic 



(N=12) (N=7) (N=32) 


Source: Interviews with state administrators in spring 2013. 

Note: The policy or practice summarized in this figure is presented in Table IV. 5. Each dot in this figure 

represents the states that reported using the one policy or practice that was aligned with the criterion in the 
RTT application about intervening in the lowest-achieving schools. The number inside each dot is the 
number of states represented by the dot; dots that represent only one state have no number inside. The 
dashed line denotes the average number of policies and practices for each group of states. There were no 
statistically significant differences between RTT and non-RTT states at the 0.05 level using a two-tailed 
test. 

LEA = local education agency. 
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Figure A.14. Use of policies and practices aligned with RTT, turning around 
the lowest-achieving schools subtopic 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV.5. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of nine 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For five of the 
policies and practices, a “yes” response received one point. In the other cases, it was possible for a state to 
receive a fraction of one point. The dashed line denotes the average number of policies and practices for 
each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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G. Charter schools 

Figure A.15. Use of policies and practices aligned with RTT, eliminating 
restrictions on charter school creation and enrollment subtopic 



Source: Interviews with state administrators in spring 2013. 

Note: The policy or practice summarized in this figure is presented in Table IV. 6. Each dot in this figure 

represents the states that reported using the one policy or practice that was aligned with the RTT 
application criteria. The number inside each dot is the number of states represented by the dot; dots that 
represent only one state have no number inside. The dashed line denotes the average number of policies 
and practices for each group of states. There were no statistically significant differences between RTT and 
non-RTT states at the 0.05 level using a two-tailed test. 
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Figure A.16. Use of policies and practices aligned with RTT, refining 
authorization and monitoring processes for charter schools subtopic 



Source: Interviews with state administrators in spring 2013. 

Note: The policies and practices summarized in this figure are presented in Table IV. 6. Each dot in this figure 

represents the states that reported using a particular number of policies and practices (out of three 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For two of the 
policies and practices, a “yes” response received one point. In the third case, it was possible for a state to 
receive a fraction of one point. The dashed line denotes the average number of policies and practices for 
each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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In the main body of the report, we summarized the extent to which states reported using the 
policies and practices promoted by RTT, including the individual policies and practices within 
each topic area and subtopic. As described in Appendix A, some policies or practices were 
comprised of multiple interview questions. In this appendix, we present detailed findings from 
particular individual interview questions and describe how we analyzed those data. Specifically, 
we show the number of early RTT states, later RTT states, and non-RTT states that responded 
“yes” to each interview question examined as part of this report. Readers interested in states’ 
responses to individual interview questions may find this appendix useful. 

In Section A, we discuss how we analyzed data from closed- and open-ended questions and 
how we handled missing values. In Section B, we present findings from the interview questions 
in a series of tables, the titles of which are shown in the list of tables at the beginning of this 
report. The tables in Section B show data from both the spring 2012 and spring 2013 interviews. 
Because Texas did not participate in the spring 2012 interview, it is excluded from the tables so 
that the number of states is comparable across the two years. This means the spring 2013 results 
in this appendix are not directly comparable to the spring 2013 results presented in Chapters IV 
and V (which include Texas). Two of the tables in Section B involve data from sources other 
than the state interviews described in Chapter II: in Table B.48, we present data from the 
Common Core of Data; and in Table B.59, we present data from the National Alliance for Public 
Charter Schools. 

A. Analysis methods 

Analyzing data from closed-ended questions. The evaluation’s protocol for interviews 
with state administrators comprised mostly closed-ended questions — that is, questions with “yes” 
or “no” responses or with a set of specific response categories from which to choose. As a result, 
these variables were already in a format that is suitable, or nearly suitable, for analysis. 

Closed-ended questions sometimes included an “other-specify” response option so the 
interview could progress smoothly when a respondent was uncertain about the response option 
that applied or could not find a response option that adequately captured the response he or she 
wished to provide. When a respondent chose this option, the interviewer asked the respondent to 
specify his or her response and recorded it. These “other-specify” responses were reviewed and 
either recoded into one of the existing structured response categories or coded into a new 
response category, as appropriate. Following reporting requirements established by the U.S. 
Department of Education’s National Center for Education Statistics, we created a new response 
category only if at least three respondents (that is, states) provided the same or similar response. 
If fewer than three respondents provided a particular response, the response remained part of the 
broad “other” category. 

Analyzing data from open-ended questions. Whenever possible, we categorized the 
responses to open-ended questions into nominal categories (based on the themes that emerged) 
that could then be treated as quantitative, categorical data. This strategy enabled us to 
systematically identify and report on recurring themes mentioned frequently by respondents. 

Handling missing values. Values can be missing for various reasons: (1) because the 
respondent did not complete the interview; (2) because the respondent completed the interview 
but did not complete the question; (3) because the respondent chose “don’t know,” “refused,” or 
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“not applicable”; or (4) because the question was logically skipped based on earlier responses. 
Generally, we excluded all missing values from our calculations regardless of the reason that the 
question was missing (that is, we did not recode a missing as a zero). In the tables, we report the 
sample sizes for states with nonmissing values on the given item. Frequencies generally total 50 
(the number of states that completed the interview; Texas did not participate in the spring 2012 
interview and is therefore excluded from this appendix). 

Selecting interview questions aligned with the RTT application criteria. As described in 
Chapter II and Appendix A, we reviewed the interview questions and assigned those that aligned 
with the policies and practices described in the RTT application selection criteria to specific 
topic areas and subtopics. We determined the subtopic into which each interview question fell 
based on the section of the RTT application criteria with which it aligned. In the tables presented 
in Section B, the last column of each table indicates whether each question was selected, and if it 
was selected, for which subtopic, by using the abbreviations shown in Table B.l. 


Table B.l. Abbreviations for subtopics 


Subtopic 

Abbreviation 

Topic Area: State Capacity 



Articulating the state’s education reform agenda and LEAs’ participation in it SC-1 

Building strong statewide capacity to implement, scale up, and sustain the proposed plans SC-2 

Topic Area: Standards and Assessments 

Developing and adopting common standards SA-1 

Developing and implementing common, high-quality assessments SA-2 

Supporting the transition to enhanced standards and high-quality assessments SA-3 

Topic Area: Data Systems and Use of Data to Improve Instruction 

Fully implementing a statewide longitudinal data system 
Accessing state data and using it to inform key stakeholders 
Using data to improve instruction 

Topic Area: Teacher and Principal Certification and Evaluation 

Providing high-quality pathways to certification for aspiring teachers and principals 
Improving teacher and principal effectiveness based on performance 
Ensuring equitable distribution of effective teachers and principals 
Improving the effectiveness of teacher and principal preparation programs 



Eliminating restrictions on charter school creation and enrollment 
Refining authorization and monitoring processes for charter schools 

Source: Interviews with state administrators in spring 2012 and spring 2013. 

LEAs = local education agencies. 

B. Detailed findings from interview questions 

In this section we present findings from particular interview questions based on U.S. 
Department of Education guidance about those in which it had the most interest. The tables are 
organized to follow the order of the modules in the state interview protocol, which was: (1) state 
capacity, (2) standards and assessments, (3) data systems, (4) teachers and leaders, (5) school 
turnaround, and (6) charter schools. 


CH-1 

CH-2 


TL-1 

TL-2 

TL-3 

TL-4 


DA-1 

DA-2 

DA-3 
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Table B.2. State implementation of comprehensive education reform plans 


- 

Early RTT 

Number of states 

Later RTT 

Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Reported having a comprehensive education 
reform plan in place 

2011-2012 

12 

7 

27 

Yes (SC-1) 

2012-2013 

12 

7 

27 


Reported using the following strategies to 
implement this reform plan: 

Working with districts that have the lowest 
achievement levels to improve their 
performance 

2011-2012 

12 

7 

26 

No 

.2012-2013 

12 

7 

26 


Working with districts that have the highest 
achievement levels and seeking to replicate 
their practices statewide 

2011-2012 

11 

4 

12 

Yes (SC-2) 

2012-2013 

10 

5 

15 


Working with a specific set of districts that are 
not necessarily the highest- or lowest-achieving 
in the state 

2011-2012 

12 

6 

20 

No 

2012-2013 

11 

7 

20 


Other strategy 

2011-2012 

3 

3 

12 

No 

2012-2013 

3 

6 

13 


Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


SC-1 = Articulating the State’s education reform agenda and LEAs’ participation in it; SC-2 = Building strong 
statewide capacity to implement, scale up, and sustain the proposed plans. 
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Table B.3. States’ education reform priorities 






Number of states 


Item aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

application 

(subtopic) 

Reported prioritizing the following education 
reform initiatives to a great extent: 

Adopting and implementing college- and career- 




Yes (SC-1) 

ready standards and/or assessments 

2011-2012 

10 

7 

29 


2012-2013 

12 

6 

29 


Building comprehensive, student-level, longitudinal 




Yes (SC-1) 

data systems 

2011-2012 

10 

5 

24 


2012-2013 

10 

5 

23 


Using data to improve instruction 

2011-2012 

12 

6 

23 

Yes (SC-1) 

2012-2013 

10 

6 

21 


Recruiting and/or retaining effective teachers and 




Yes (SC-1) 

school leaders 

2011-2012 

7 

2 

14 


2012-2013 

6 

3 

10 


Rewarding effective teachers and school leaders 
2011-2012 

3 

1 

6 

Yes (SC-1) 

2012-2013 

1 

1 

3 


Developing and preparing effective teachers and 




Yes (SC-1) 

school leaders 

2011-2012 

9 

4 

19 


2012-2013 

11 

4 

10 


Developing and implementing a teacher and 




Yes (SC-1) 

principal evaluation system that is based on 
student growth 

2011-2012 

11 

7 

24 


2012-2013 

10 

7 

22 


Improving the distribution of effective teachers and 




Yes (SC-1) 

principals 

2011-2012 

5 

2 

6 


2012-2013 

3 

2 

3 


Turning around the lowest-achieving schools 
2011-2012 

12 

6 

24 

Yes (SC-1) 

2012-2013 

12 

5 

24 


Providing supports for ELLs 

2011-2012 

4 

2 

14 

No 

2012-2013 

5 

5 

11 


Reported that ELLs fit into their current 
education reform priorities in the following 
ways: 

ELLs are an explicit, central priority of statewide 




No 

reform efforts 

2011-2012 

1 

0 

5 


2012-2013 

2 

3 

4 


ELLs are an emerging priority of statewide reform 




No 

efforts 

2011-2012 

2 

1 

8 


2012-2013 

0 

1 

9 


Statewide reform efforts are designed to address 




No 

the needs of all students, including ELLs 

2011-2012 

9 

6 

17 


2012-2013 

10 

3 

17 


Number of States 

11-12 

6-7 

28-31 
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Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

ELLs = English language learners; SC-1 = Articulating the State’s education reform agenda and LEAs’ participation in 
it. 
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Table B.4. Role of state education agency in education reform 



Early RTT 

Number of states 

Later RTT 

Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Reported that the SEA played the following 
roles to a great extent: 

Creator of a statewide vision for reforms 
2011-2012 

11 

7 

24 

Yes (SC-2) 

2012-2013 

12 

7 

26 


Creator and monitor of performance measures 
2011-2012 

11 

6 

28 

Yes (SC-2) 

2012-2013 

11 

7 

26 


Compliance monitor of reform 

2011-2012 

11 

7 

28 

Yes (SC-2) 

2012-2013 

11 

6 

27 


Facilitator between districts/schools and external 
support 

2011-2012 

1 

3 

8 

Yes (SC-2) 

2012-2013 

6 

2 

8 


Provider of direct support to districts/schools 
2011-2012 

4 

4 

15 

Yes (SC-2) 

2012-2013 

6 

3 

14 


Provider of information about federal policies 
2011-2012 

9 

7 

29 

No 

2012-2013 

10 

5 

26 


Other roles 

2011-2012 

1 

1 

2 

No 

2012-2013 

0 

1 

2 


Number of States 

12 

7 

30-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

SEA = State Education Agency; SC-2 = Building strong statewide capacity to implement, scale up, and sustain the 
proposed plans. 
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Table B.5. State provision of targeted supports for implementing statewide 
education reforms 



Number of states 

Item aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Reported providing targeted support to any 




Yes (SC-2) 

type of districts or schools 





2011-2012 

12 

7 

31 


2012-2013 

12 

7 

30 


Reported providing targeted support to the 





following groups: 





Urban districts or schools 




No 

2011-2012 

12 

5 

21 


2012-2013 

10 

5 

18 


Rural districts or schools 




No 

2011-2012 

11 

5 

25 


2012-2013 

7 

5 

22 


Districts or schools with ELLs 




No 

2011-2012 

11 

4 

28 


2012-2013 

10 

5 

24 


High-minority districts or schools 




No 

2011-2012 

11 

5 

20 


2012-2013 

9 

5 

23 


High-poverty districts or schools 




No 

2011-2012 

12 

7 

27 


2012-2013 

10 

5 

26 


Districts or schools that received SIG funds 




No 

2011-2012 

12 

7 

31 


2012-2013 

12 

7 

30 


Districts or schools “participating” in RTT a 




No 

2011-2012 

12 

6 

n.a. 


2012-2013 

11 

6 

n.a. 


Districts or schools “involved” in RTT a 




No 

2011-2012 

9 

5 

n.a. 


2012-2013 

9 

4 

n.a. 


Districts or schools identified for improvement or 




No 

corrective action 





2011-2012 

10 

7 

29 


2012-2013 

10 

7 

26 


Other types of districts or schools 




No 

2011-2012 

5 

3 

14 


2012-2013 

5 

5 

13 


Number of States 

10-12 

5-7 

28-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a These items were asked only in RTT states. 

ELLs = English language learners; SIG = School Improvement Grant; SC-2 = Building strong statewide capacity to 
implement, scale up, and sustain the proposed plans, 
n.a. = not applicable. 
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Table B.6. State monitoring of districts’ implementation of state education 
reforms 



Number of states 


Item aligned 





with RTT 





application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Reported using the following strategies to 





monitor districts’ implementation of main state 





education reforms: 





Review of district-submitted reports 




No 

2011-2012 

11 

6 

31 


2012-2013 

12 

7 

29 


Review of third-party reports 




No 

2011-2012 

11 

6 

24 


2012-2013 

11 

4 

24 


Monitoring site visits by State Department of 




No 

Education staff 





2011-2012 

12 

7 

31 


2012-2013 

12 

7 

31 


Examination of student assessment data by State 




No 

Department of Education staff 





2011-2012 

12 

7 

30 


2012-2013 

12 

7 

31 


Other strategies 




No 

2011-2012 

2 

2 

8 


2012-2013 

4 

4 

8 


Reported taking the following actions in 





response to monitoring results 





Provided additional resources 




No 

2011-2012 

11 

7 

28 


2012-2013 

12 

7 

28 


Revised plans for/expectations of districts 




No 

2011-2012 

11 

7 

31 


2012-2013 

11 

7 

29 


Removed state funding/supports 




No 

2011-2012 

5 

2 

8 


2012-2013 

4 

2 

4 


Provided targeted support 




No 

2011-2012 

12 

7 

31 


2012-2013 

12 

6 

30 


Other actions 




No 

2011-2012 

2 

2 

7 


2012-2013 

0 

4 

4 


Number of States 

12 

7 

31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 
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Table B.7. State work with intermediaries to support implementation of 

statewide education reforms 







Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Reported working with intermediaries to 
support the implementation of statewide 
education reforms 




Yes (SC-2) 

2011-2012 

12 

7 

31 


2012-2013 

10 

6 

30 


Reported working with the following types of 
intermediaries to support the implementation of 
statewide education reforms: 

Federally supported comprehensive center, 
regional educational laboratory, equity assistance 
center, or content center 




No 

2011-2012 

11 

6 

31 


2012-2013 

Institution of higher education 

8 

5 

30 

No 

2011-2012 

11 

6 

30 


2012-2013 

Regional/county offices 

10 

5 

28 

No 

2011-2012 

7 

4 

14 


2012-2013 

5 

4 

13 


Educators contracted by the state, such as 
distinguished educators 




No 

2011-2012 

9 

6 

25 


2012-2013 

Other external organizations 

8 

2 

18 

No 

2011-2012 

8 

3 

21 


2012-2013 

Other types of intermediary 

7 

4 

17 

No 

2011-2012 

1 

1 

6 


2012-2013 

3 

4 

5 


Number of States 

8-12 

6-7 

19-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

SC-2 = Building strong statewide capacity to implement, scale up, and sustain the proposed plans. 
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Table B.8. Groups with which states expected intermediaries to work 

- 

Early RTT 

Number of states 

Later RTT 

Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Reported intermediaries were expected to 
work with the following groups: 

State-level staff 

2011-2012 

12 

7 

30 

No 

2012-2013 

10 

6 

30 


All districts 

2011-2012 

7 

4 

18 

No 

2012-2013 

6 

5 

14 


Districts identified for improvement or corrective 
action under NCLB 

2011-2012 

8 

5 

25 

No 

2012-2013 

5 

5 

22 


Schools identified for improvement, corrective 
action, or restructuring under NCLB and/or the 
districts in which these schools are located 
2011-2012 

10 

5 

25 

No 

2012-2013 

7 

4 

25 


Schools that received SIG or RTT funds to 
implement one of the four U.S. Department of 
Education-specified school intervention models 
and/or the districts in which these schools are 
located 

2011-2012 

12 

6 

28 

No 

2012-2013 

10 

5 

27 


Districts and/or schools that are formally 
defined as “participating” in RTT a 

2011-2012 

12 

3 

n.a. 

No 

2012-2013 

10 

3 

n.a. 


Districts and/or schools that are formally 
defined as being “involved” in RTT a 

2011-2012 

7 

3 

n.a. 

No 

2012-2013 

7 

2 

n.a. 


Other groups of districts and/or schools 
2011-2012 

2 

2 

5 

No 

2012-2013 

4 

2 

6 


Number of States 

9-12 

4-7 

28-31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a These items were asked only in RTT states. 

NCLB = No Child Left Behind; SIG = School Improvement Grant, 
n.a. = not applicable. 
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Table B.9. Structural changes at the state education agency 


_ 


Number of states 


Item aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

application 

(subtopic) 

Reported implementing the following structural 
changes at the SEA since July 2008/spring 2012 
to support implementation of state education 
reforms: 

Created new office(s) or department(s) 

2011-2012 

11 

5 

15 

No 

2012-2013 

4 

3 

9 


Created new staff positions 

2011-2012 

12 

7 

25 

No 

2012-2013 

11 

6 

15 


Reorganized the structure of existing 
offices/departments 

2011-2012 

11 

7 

24 

No 

2012-2013 

6 

7 

18 


Changed laws, policies, or regulations 

2011-2012 

11 

7 

26 

No 

2012-2013 

8 

5 

21 


Other structural changes to support the 
implementation of reforms 

2011-2012 

3 

2 

10 

No 

2012-2013 

0 

2 

8 


Reported the following changes since July 
2008/spring 2012 to improve the SEA’s capacity 
to address the needs of ELLs: 

Increased SEA staff with ELL expertise 

2011-2012 

2 

2 

11 

No 

2012-2013 

3 

3 

8 


Reorganized structure of existing offices/ 
departments 

2011-2012 

12 

4 

23 

No 

2012-2013 

7 

5 

19 


Increased use of consultants with ELL expertise 
2011-2012 

9 

3 

20 

No 

2012-2013 

6 

3 

10 


Increased ELL expertise within regional offices 
2011-2012 

6 

3 

13 

No 

2012-2013 

3 

2 

4 


Redefined policies to better meet ELLs’ needs 
2011-2012 

7 

3 

21 

No 

2012-2013 

7 

1 

11 


Other types of changes 

2011-2012 

4 

2 

6 

No 

2012-2013 

2 

2 

4 


Number of States 

8-12 

6-7 

18-31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


SEA = State Education Agency; ELL = English language learners. 
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Table B.10. State coordination of RTT and/or SIG with other federal programs 


Number of states Item aligned 

with RTT 
application 

Early RTT Later RTT Non-RTT (subtopic) 


Reported coordinating RTT and/or SIG with 
other federal programs in the following 
ways since July 2008/spring 2012: 

Encouraged cofunding of initiatives using 
funds from different federal programs 

2011- 2012 12 7 

2012- 2013 12 7 

Established common planning activities for 

state Department of Education staff 
responsible for different federal programs 

2011- 2012 11 7 

2012- 2013 10 7 

Increased communication across state 

Department of Education staff responsible for 
different federal programs 

2011- 2012 12 7 

2012- 2013 12 7 

Other steps to coordinate RTT and/or SIG and 

other federal programs 

2011- 2012 5 4 

2012- 2013 7 3 

Number of States 12 7 


No 


27 

28 

No 


27 

29 

No 


28 

31 


14 

13 

31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 
SIG = School Improvement Grant. 
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Table B.11. State-level gaps in expertise to support education reforms 


- 

Early RTT 

Number of states 

Later RTT 

Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Reported gaps in the following areas of 
expertise: 

Monitoring districts and/or schools 




No 

2011-2012 

1 

1 

6 


2012-2013 

Providing targeted support or technical assistance 

2 

1 

7 

No 

to districts and/or schools 

2011-2012 

2 

2 

11 


2012-2013 

Developing and working with districts to 

3 

3 

14 

No 

implement teacher evaluation models 

2011-2012 

0 

3 

10 


2012-2013 

Developing state longitudinal data systems 

3 

1 

16 

No 

2011-2012 

1 

2 

6 


2012-2013 

Working with districts and/or schools on the use of 

3 

1 

9 

No 

data to improve instruction 

2011-2012 

3 

2 

9 


2012-2013 

Supporting districts and/or schools in the process 

5 

3 

12 

No 

of turning around low-achieving schools 

2011-2012 

3 

2 

13 


2012-2013 

Other types of expertise 

4 

3 

15 

No 

2011-2012 

2 

1 

4 


2012-2013 

4 

2 

10 


Number of States 

12 

7 

29-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 
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Table B.12. Barriers constraining state educational agency capacity to 
implement statewide education reforms 


Number of states Item aligned 



Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Reported the following factors as most 
constraining SEA’s capacity to implement 
statewide education reforms: 

Too few SEA staff 

2011-2012 

3 

0 

9 

No 

2012-2013 

4 

1 

8 


SEA staff working in silos 

2011-2012 

2 

0 

1 

No 

2012-2013 

1 

0 

2 


Limited expertise in reform topics 

2011-2012 

0 

1 

2 

No 

2012-2013 

0 

0 

2 


Staff turnover 

2011-2012 

1 

0 

2 

No 

2012-2013 

1 

0 

3 


Limited state funding 

2011-2012 

1 

2 

10 

No 

2012-2013 

0 

3 

5 


Limited federal funding 

2011-2012 

0 

1 

0 

No 

2012-2013 

0 

1 

1 


Need for better technology to support reform efforts 
2011-2012 

0 

0 

0 

No 

2012-2013 

2 

1 

0 


Provisions within collective bargaining agreements 
2011-2012 

2 

0 

1 

No 

2012-2013 

1 

0 

1 


State regulations/laws 

2011-2012 

1 

1 

2 

No 

2012-2013 

1 

0 

4 


Federal regulations/laws 

2011-2012 

0 

0 

1 

No 

2012-2013 

1 

1 

0 


Governance structure of state 

2011-2012 

0 

0 

0 

No 

2012-2013 

1 

0 

2 


Level of support from governor 

2011-2012 

0 

0 

0 

No 

2012-2013 

0 

0 

0 


Level of state legislature support 

2011-2012 

0 

0 

1 

No 

2012-2013 

0 

0 

1 


Other factors 

2011-2012 

2 

2 

2 

No 

2012-2013 

0 

0 

2 


Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 
SEA = State Education Agency. 
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Table B.13. State adoption of the Common Core State Standards 




Number of states 


Item aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

application 

(subtopic) 

Reported adopting the CCSS in the following 
subject areas: 

Both ELA and math 

2011-2012 

11 

7 

25 

Yes (SA-1) 

2012-2013 

12 

6 

27 


Math only 

2011-2012 

0 

0 

0 

Yes (SA-1) 

2012-2013 

0 

0 

0 


ELA only 

2011-2012 

1 

0 

3 

Yes (SA-1) 

2012-2013 

0 

1 

1 


Reported supplementing the CCSS with state- 
specific standards in the following subject 
areas: 3 

Both ELA and math 

2011-2012 

2 

2 

5 

Yes (SA-1) 

2012-2013 

1 

2 

7 


Math only 

2011-2012 

0 

0 

0 

Yes (SA-1) 

2012-2013 

0 

0 

0 


ELA only 

2011-2012 

1 

1 

2 

Yes (SA-1) 

2012-2013 

1 

1 

2 


Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


a States that reported they did not adopt the CCSS in either Math or English are included in the analysis of this 
question as no responses. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-1 = Developing and adopting common 
standards. 
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Table B.14. Changes to state policies and practices in 
the Common Core State Standards 

response to adopting 


Among the states that adopted the 



CCSS in ELA and/or math, number of 

Item aligned 



states 


with RTT 





application 


Early RTT 

Later RTT Non-RTT 

(subtopic) 

Reported the following changes to state policies 





and practices in response to adopting the CCSS: 





Required use of new curricula or textbooks 




Yes (SA-3) 

2011-2012 

0 

0 

4 


2012-2013 

3 

1 

4 


Required use of new, state-approved interim 




Yes (SA-3) 

assessments 





2011-2012 

1 

0 

0 


2012-2013 

2 

0 

2 


Changed credit or course requirements 




Yes (SA-3) 

2011-2012 

3 

1 

3 


2012-2013 

2 

2 

4 


Changed the content of the state’s high school exit 




Yes (SA-3) 

exam 





2011-2012 

0 

1 

1 


2012-2013 

1 

1 

1 


Changed the performance standard on the high 




Yes (SA-3) 

school exit exam 





2011-2012 

0 

0 

1 


2012-2013 

1 

0 

0 


Changed the college entrance requirements for the 




Yes (SA-3) 

state college/university system 





2011-2012 

0 

0 

2 


2012-2013 

1 

0 

1 


Changed policies related to teachers and/or school 




Yes (SA-3) 

leaders, such as licensure, certification, or annual 





professional development requirements 





2011-2012 

5 

0 

6 


2012-2013 

7 

3 

9 


Other changes 




No 

2011-2012 

5 

2 

10 


2012-2013 

3 

1 

8 


Number of States 

6-12 

4-7 

12-28 


Source: Interviews with state administrators in spring 2012 and spring 2013. 



Note: A range is provided for the sample sizes because nonresponse varied across items. 


CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition to enhanced 

standards and high-quality assessments. 
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Table B.15. State participation in consortia to develop assessments aligned 
to Common Core State Standards 



Among the states that adopted the CCSS 
in ELA and/or math, number of states 

Item aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

application 

(subtopic) 

Reported participating in a consortium to 




Yes (SA-2) 

develop assessments aligned to these 
standards 

2011-2012 

12 

7 

27 


2012-2013 

12 

7 

25 


Reported participating in the following 
consortia: 

Partnership for Assessment of Readiness for 




No 

College and Careers (PARCC) only 

2011-2012 

9 

5 

5 


2012-2013 

SMARTER Balanced Assessment Consortium 

9 

6 

5 

No 

(SBAC) only 

2011-2012 

3 

0 

19 


2012-2013 

Both PARCC and SBAC 

3 

0 

17 

No 

2011-2012 

0 

2 

2 


2012-2013 

Other consortium 

0 

1 

1 

No 

2011-2012 

0 

0 

1 


2012-2013 

0 

0 

2 


Number of States 

12 

7 

28 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-2 = Developing and implementing 
common, high-quality assessments. 


B-18 




APPENDIX B 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.16. Funds provided by states to districts and/or schools to support 
implementation of Common Core State Standards 


Among the states that adopted the CCSS in 

Item aligned 


ELA and/or math, number of states 

with RTT 


Early RTT 

Later RTT 

Non-RTT 

application 

(subtopic) 

Reported providing funds to districts 




Yes (SA-3) 

and/or schools to support implementation 
of the CCSS since adopting them 

2011-2012 

8 

1 

7 


2012-2013 

9 

1 

10 


Reported designating such funds for: 

Professional development 

2011-2012 

8 

1 

5 

Yes (SA-3) 

2012-2013 

9 

1 

10 


Curriculum/textbook purchase 

2011-2012 

3 

0 

2 

Yes (SA-3) 

2012-2013 

5 

0 

2 


Development or purchase of interim 




Yes (SA-3) 

assessments or item banks 

2011-2012 

5 

0 

1 


2012-2013 

7 

1 

5 


Purchase of hardware or software needed to 




Yes (SA-3) 

implement standards or assessments 

2011-2012 

4 

0 

3 


2012-2013 

6 

0 

2 


Fund additional staff either externally 




Yes (SA-3) 

contracted or hired internally 

2011-2012 

6 

0 

2 


2012-2013 

7 

0 

2 


Other purpose 

2011-2012 

4 

0 

1 

No 

2012-2013 

3 

1 

2 


Number of States 

12 

7 

26 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition to enhanced 
standards and high-quality assessments. 
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Table B.17. State investments in new technology to assist with 
implementation of the Common Core State Standards 


Among the states that adopted the CCSS in Item aligned 
ELA and/or math, number of states with RTT 

application 

Early RTT Later RTT Non-RTT (subtopic) 


Reported investing in new technology to 
assist with implementation of the CCSS 
since adopting them 

2011- 2012 7 3 

2012- 2013 10 2 

Reported the following technology 
investments for this purpose: 

Hardware 

2011-2012 6 0 

2012-2013 8 1 

Software 

2011-2012 6 2 

2012-2013 9 2 

Connectivity, such as increased bandwidth or 
Internet speed for school systems 

2011- 2012 5 1 

2012- 2013 6 1 

Other investments 

2011-2012 2 2 

2012-2013 2 0 

Number of States 12 7 


Yes (SA-3) 


7 

8 


4 

5 

4 

4 


No 


No 


No 


4 

5 

4 

_4_ 

27 


No 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition to enhanced 
standards and high-quality assessments. 
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Table B.18. State investments in new technology to assist with 
implementation of the assessments associated with the Common Core State 
Standards 


Among the states that adopted the CCSS Item aligned 
in ELA and/or math, number of states with RTT 

application 

Early RTT Later RTT Non-RTT (subtopic) 


Reported investing in new technology to 
assist with implementation of the 
assessments associated with the CCSS since 


adopting them 

2011-2012 6 

2012-2013 7 

Reported the following technology 
investments for this purpose: 

Hardware 

2011- 2012 5 

2012- 2013 6 

Software 

2011-2012 6 

2012-2013 5 

Connectivity, such as increased bandwidth or 
Internet speed for conducting assessments 

2011- 2012 5 

2012- 2013 5 

Other investments 

2011-2012 0 

2012-2013 2 

Number of States 12 


Yes (SA-3) 


2 4 

2 9 


0 

1 

2 

2 


2 

4 

3 

5 


No 


No 


No 


0 

1 

1 

0 

6-7 


3 

5 

1 

J_ 

27 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition to enhanced 
standards and high-quality assessments. 
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Table B.19. Materials provided by states to help practitioners understand the 
Common Core State Standards and/or change instruction based on the 
standards 


Among the states that adopted the CCSS Item aligned 
in ELA and/or math, number of states with RTT 



Early RTT 

Later RTT 

Non-RTT 

application 

(subtopic) 

Reported providing materials to help 
practitioners understand the CCSS and/or 
change instruction based on the standards 
since adoption of these standards 

2011-2012 

12 

7 

28 

Yes (SA-3) 

2012-2013 

12 

7 

27 


Reported providing the following types of 
materials for this purpose: 

Documents showing alignment between the 
previous state standards and the new standards 
2011-2012 

10 

7 

26 

No 

2012-2013 

8 

5 

21 


Documents showing the alignment between 
summative assessments and new standards, 
such as blueprints 

2011-2012 

4 

4 

11 

No 

2012-2013 

8 

7 

18 


Tools or guidance on providing instruction aligned 
with the new standards, such as scope and 
sequence, curriculum maps, or frameworks 
2011-2012 

11 

7 

21 

No 

2012-2013 

11 

7 

23 


Examples of instruction consistent with new 
standards 

2011-2012 

10 

5 

22 

No 

2012-2013 

11 

4 

26 


Sample lesson plans consistent with new 
standards 

2011-2012 

6 

3 

18 

No 

2012-2013 

9 

5 

20 


Sample performance tasks for formative 
assessment purposes, including rubrics or 
scoring guides 

2011-2012 

7 

2 

11 

No 

2012-2013 

11 

3 

18 


Sample student work 

2011-2012 

4 

2 

7 

No 

2012-2013 

6 

2 

10 


Banks of interim assessment items aligned to 
standards 

2011-2012 

2 

3 

4 

No 

2012-2013 

6 

4 

10 


Walk-through or observation protocols to aid in 
monitoring alignment of instruction to new 
standards 

2011-2012 

7 

2 

5 

No 

2012-2013 

9 

4 

10 


Other types of materials 

2011-2012 

6 

0 

17 

No 

2012-2013 

6 

3 

10 


Number of States 

12 

6-7 

27-28 
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Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition to enhanced 
standards and high-quality assessments. 


B-23 



APPENDIX B 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.20. State support to districts and/or schools for implementation of 
Common Core State Standards 


Among the states that adopted the CCSS Item aligned 
in ELA and/or math, number of states with RTT 



Early RTT 

Later RTT 

Non-RTT 

~ application 
(subtopic) 

Reported providing professional 
development, training, or technical assistance 
to districts and/or schools to support 
implementation of the CCSS since adoption of 
these standards 

2011-2012 

12 

7 

27 

Yes (SA-3) 

2012-2013 

12 

7 

28 


Reported providing the following types of 
support for this purpose: 

Training on building awareness/understanding of 
the standards 

2011-2012 

12 

7 

27 

No 

2012-2013 

12 

7 

28 


Training on teaching strategies, activities, or 
methods aligned with the standards 

2011-2012 

12 

5 

25 

No 

2012-2013 

12 

5 

27 


Training on the assessments aligned to the 
standards 

2011-2012 

6 

4 

17 

No 

2012-2013 

7 

5 

21 


Other types of support 

2011-2012 

4 

0 

7 

No 

2012-2013 

6 

3 

9 


Number of States 

12 

7 

28 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition to enhanced 
standards and high-quality assessments. 
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Table B.21. State support to districts and/or schools to aid in implementation 
of Common Core State Standards with English language learners 


Among the states that adopted the CCSS Item aligned 
in ELA and/or math, number of states with RTT 

application 

Early RTT Later RTT Non-RTT (subtopic) 


Reported providing supports to districts 
and/or schools specifically designed to aid in 


the implementation of the CCSS with ELLs 
since adopting these standards 

2011-2012 11 

2012-2013 12 

Reported providing the following support for 
this purpose: 

Funds 

2011- 2012 4 

2012- 2013 7 

Materials to support understanding and use of 

new standards with ELLs 

2011- 2012 7 

2012- 2013 11 

Professional development, training, or technical 
assistance to support the instruction of ELLs 

2011-2012 8 

2012-2013 12 

Mapped new ELA standards to state English 
language proficiency standards or revised state 
English language proficiency standards for better 
alignment 

2011- 2012 5 

2012- 2013 6 

Other support 

2011-2012 1 

2012-2013 1 

Number of States 11-12 


Yes (SA-3) 


18 

23 


4 

3 


15 

19 


15 

23 


No 


No 


No 


No 


15 

12 

4 

4 


No 


26-27 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

CCSS = Common Core State Standards; ELA = English/language arts; ELLs = English language learners; SA-3 = 
Supporting the transition to enhanced standards and high-quality assessments. 
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Table B.22. State implementation of Statewide Longitudinal Data Systems 

— 

Early RTT 

Number of states 

Later RTT 

Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Reported having an SLDS 

2011-2012 

12 

7 

28 

Yes (DA-1) 

2012-2013 

12 

7 

30 


Reported that the following data systems 
are linked to the state’s K-12 education 
longitudinal data system: 

Early childhood 

2011-2012 

10 

5 

15 

Yes (DA-1 ) 

2012-2013 

11 

5 

19 


Higher education 

2011-2012 

9 

5 

18 

Yes (DA-1) 

2012-2013 

10 

7 

23 


Teacher preparation programs 

2011-2012 

4 

2 

13 

No 

2012-2013 

7 

4 

8 


Health (e.g., access to medical or mental 
health services) 

2011-2012 

3 

2 

5 

No 

2012-2013 

4 

1 

4 


Welfare/foster care/child protective services 
2011-2012 

3 

0 

12 

No 

2012-2013 

5 

3 

10 


Workforce (e.g., employment data) 

2011-2012 

3 

0 

9 

No 

2012-2013 

4 

2 

9 


Juvenile justice 

2011-2012 

3 

3 

10 

No 

2012-2013 

4 

4 

8 


Other data systems 

2011-2012 

3 

1 

8 

No 

2012-2013 

2 

1 

10 


Number of States 

11-12 

7 

30-31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


SLDS = Statewide Longitudinal Data System; DA-1 

= Fully implementing a statewide longitudinal data system. 
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Table B.23. State requirements related to district data systems 


Number of states 


Item aligned 

- 



with RTT 




application 


Early RTT Later RTT 

Non-RTT 

(subtopic) 

Reported requiring districts to implement district 



Yes (DA-3) 

data systems 3 




2011-2012 

6 1 

8 


2012-2013 

6 3 

11 


Number of States 

12 7 

31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

a District data systems (also identified as local instructional improvement systems) are defined in the RTT application 
as technology-based tools or other strategies that provide teachers, principals, and administrators with meaningful 
support and actionable data to systemically manage continuous instructional improvement, including such activities 
as instructional planning; gathering information (for example, through formative, interim, and summative 
assessments, and looking at student work and other student data); analyzing information with the support of rapid- 
time reporting; using this information to inform decisions on appropriate next instructional steps; and evaluating the 
effectiveness of the actions taken. Such systems promote collaborative problem solving and action planning; they 
may also integrate instructional data with student-level data, such as attendance, discipline, grades, credit 
accumulation, and student survey results, to provide early-warning indicators of a student’s risk of educational failure. 
DA-3 = Using data to improve instruction. 
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Table B.24. Access to State Longitudinal Data Systems 



Among the states that reported having a 
Statewide Longitudinal Data System, number of 
states 

Early RTT Later RTT Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Reported that the following groups 
had access to data from the SLDS: 

State Department of Education staff in: 
2007-2008 3 

8 

5 

19 

Yes (DA-2) 

2011-2012 

8 

6 

20 


2012-2013 

8 

6 

20 


District staff in: 

2007-2008 3 

8 

4 

14 

Yes (DA-2) 

2011-2012 

8 

5 

20 


2012-2013 

7 

5 

19 


Principals in: 

2007-2008 3 

7 

3 

10 

Yes (DA-2) 

2011-2012 

8 

5 

18 


2012-2013 

7 

5 

18 


Teachers in: 

2007-2008 3 

5 

3 

8 

Yes (DA-2) 

2011-2012 

7 

5 

16 


2012-2013 

7 

4 

15 


Colleges and universities in: 

2007-2008 3 

4 

2 

7 

Yes (DA-2) 

2011-2012 

7 

5 

12 


2012-2013 

6 

4 

15 


External researchers in: 

2007-2008 3 

6 

2 

7 

Yes (DA-2) 

2011-2012 

8 

6 

14 


2012-2013 

6 

6 

17 


Other groups in: 

2007-2008 3 

1 

1 

6 

No 

2011-2012 

5 

2 

12 


2012-2013 

5 

1 

13 


Number of States 

8 

5-6 

18-20 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

SLDS = Statewide Longitudinal Data System; DA-2 = Accessing State data and using it to inform key stakeholders. 
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Table B.25. Usage of State Longitudinal Data Systems 


Number of states 

Item aligned 



with RTT 



application 

Early RTT 

Later RTT Non-RTT 

(subtopic) 


Among the states that reported having an SLDS No 

as of spring 2012/2013, reported tracking 
overall usage of their SLDS 


2011-2012 

8 

4 

15 


2012-2013 

8 

4 

16 


Among states that reported having an SLDS as 
of spring 2012/2013, reported tracking usage of 
the SLDS by user type 

2011-2012 

4 

2 

5 

No 

2012-2013 

4 

3 

4 


Among the states that reported tracking usage 
of their SLDS by user type, mean percentage of 
users of each type who accessed the system in 
each of the following school years: 

State Department of Education staff 

2010-2011 

20.2 

12.2 

46.6 

No 

2011-2012 

26.8 

11.8 

52.2 


District staff 

2010-2011 

20.0 

NA 

43.8 

No 

2011-2012 

62.7 

NA 

65.7 


Principals 

2010-2011 

57.0 

66.9 

43.8 

No 

2011-2012 

62.7 

59.6 

56.3 


Teachers 

2010-2011 

35.9 

100.0 

22.8 

No 

2011-2012 

29.4 

66.8 

19.1 


Number of States 

2-12 

0-7 

1-27 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

SLDS = Statewide Longitudinal Data System. 

NA = not available. 
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Table B.26. Uses of statewide longitudinal data by state staff 


Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Among states that reported having an SLDS, 
reported state-level staff use of data in the SLDS 
for the following purposes: 

To track overall school performance and identify 
areas for improvement 




Yes (DA-2) 

2011-2012 

11 

6 

24 


2012-2013 

12 

7 

25 


To evaluate instructional programs, such as 
measuring program effectiveness 




Yes (DA-2) 

2011-2012 

10 

5 

16 


2012-2013 

11 

7 

23 


To inform professional development offerings, such as 
identifying specific content or skills where teachers 
need assistance or support 




Yes (DA-2) 

2011-2012 

6 

1 

10 


2012-2013 

7 

4 

11 


To evaluate the success of professional development 
offerings for teachers or principals 




Yes (DA-2) 

2011-2012 

3 

1 

6 


2012-2013 

To inform resource allocation, such as which schools 
and students receive which programs or which staff 
work with which students 

3 

3 

5 

Yes (DA-2) 

2011-2012 

7 

4 

13 


2012-2013 

10 

5 

16 


To provide information to teachers about their 
students’ progress 




Yes (DA-2) 

2011-2012 

8 

5 

17 


2012-2013 

5 

7 

18 


To provide information to parents about the school or 
their children 




Yes (DA-2) 

2011-2012 

7 

5 

14 


2012-2013 

To provide information to students about their own 
progress 

6 

5 

16 

Yes (DA-2) 

2011-2012 

3 

3 

6 


2012-2013 

2 

1 

5 


To track students’ postsecondary enrollment and 
progress after high school graduation, such as credits 
earned in public colleges or universities in the state 




Yes (DA-2) 

2011-2012 

7 

5 

14 


2012-2013 

9 

6 

18 


To provide information to federal agencies (e.g., 
EDFacts) 




Yes (DA-2) 

2011-2012 

10 

6 

23 


2012-2013 

Other purposes 

12 

7 

23 

No 

2011-2012 

3 

2 

8 


2012-2013 

3 

1 

9 


Number of States 

11-12 

6-7 

26-27 


Source: Interviews with state administrators in spring 2012 and spring 2013. 



Note: A range is provided for the sample sizes because nonresponse varied across items. 


SLDS = Statewide Longitudinal Data System; DA-2 = Accessing State data and using it to inform key stakeholders. 
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Table B.27. State support for district data use 





Number of states 

i 

Item aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Provided funding, materials, training, 
technical assistance, or other supports to 
districts to encourage use of data to improve 
instruction 

2011-2012 

12 

4 

29 

Yes (DA-3) 

2012-2013 

12 

7 

29 


Provided the following supports to districts to 
encourage use of data to improve instruction: 

Funds 

2011-2012 

10 

3 

12 

No 

2012-2013 

11 

5 

11 


Materials or documents on use of data to improve 
instruction 

2011-2012 

11 

4 

23 

No 

2012-2013 

12 

7 

26 


Technical assistance 

2011-2012 

10 

3 

29 

No 

2012-2013 

12 

6 

27 


Professional development or training 

2011-2012 

10 

3 

24 

No 

2012-2013 

12 

7 

26 


Other types of support 

2011-2012 

5 

1 

9 

No 

2012-2013 

3 

0 

6 

No 

Provided funding, materials, training, 
technical assistance, or other supports to 
districts specifically to aid in use of data 
related to ELLs 

2011-2012 

7 

6 

17 

2012-2013 

10 

5 

13 


Provided following supports to districts to aid 
in use of data related to ELLs 

Funds 

2011-2012 

2 

2 

7 

No 

2012-2013 

5 

2 

5 


Materials or resources such as documents or 
software 

2011-2012 

4 

5 

15 

No 

2012-2013 

6 

3 

10 


Professional development, training, or technical 
assistance 

2011-2012 

7 

6 

17 

No 

2012-2013 

10 

5 

11 


Other types of support 

2011-2012 

1 

3 

3 

No 

2012-2013 

1 

0 

3 



Number of States 10-12 7 29-31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

ELLs = English language learners; DA-3 = Using data to improve instruction. 
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Table B.28. Access to individual student-level data in State Longitudinal Data 
Systems 


Item aligned 
with RTT 
application 
(subtopic) 


Reported that the following groups have 
access to individual student-level data for 
students in the state from the SLDS: 


State Department of Education staff 
2011-2012 

12 

7 

24 

No 

2012-2013 

12 

6 

24 


District staff 

2011-2012 

12 

7 

26 

No 

2012-2013 

10 

6 

24 


Principals 

2011-2012 

10 

5 

23 

No 

2012-2013 

8 

6 

22 


Teachers 

2011-2012 

9 

4 

18 

No 

2012-2013 

7 

4 

19 


Colleges and universities 

2011-2012 

3 

2 

5 

No 

2012-2013 

3 

3 

9 


External researchers 

2011-2012 

6 

5 

11 

No 

2012-2013 

7 

6 

20 


Other groups 

2011-2012 

0 

1 

7 

No 

2012-2013 

5 

0 

5 


Number of States 

12 

7 

25-27 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

SLDS = Statewide Longitudinal Data System. 
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Table B.29. Authorization of alternative-route certification programs for 
teachers 



Number of states 

Item aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Reported authorizing AC programs for 




Yes (TL-1 ) 

teachers in: 





2007-2008 3 

11 

6 

25 


2011-2012 

12 

7 

30 


2012-2013 

12 

7 

29 


Reported authorizing the following groups to 





operate AC programs for teachers: 





Institutions of higher education in: 




Yes (TL-1 ) 

2007-2008 3 

11 

6 

21 


2011-2012 

12 

7 

26 


2012-2013 

12 

7 

25 


School districts in: 




Yes (TL-1 ) 

2007-2008 3 

7 

2 

6 


2011-2012 

10 

3 

12 


2012-2013 

10 

4 

11 


Educational service districts or cooperatives in: 




Yes (TL-1) 

2007-2008 3 

3 

2 

5 


2011-2012 

5 

3 

9 


2012-2013 

3 

3 

10 


Nonprofit organizations in: 




Yes (TL-1 ) 

2007-2008 3 

6 

3 

10 


2011-2012 

10 

7 

18 


2012-2013 

10 

7 

16 


Partnerships or collaborations among above 




Yes (TL-1 ) 

groups in: 





2007-2008 3 

4 

4 

11 


2011-2012 

9 

6 

15 


2012-2013 

9 

5 

17 


Other groups or organizations in: 




No 

2007-2008 3 

0 

1 

10 


2011-2012 

3 

2 

14 


2012-2013 

3 

1 

12 


Number of States 

9-12 

6-7 

29-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

AC = Alternative-route certification; TL-1 = Providing high-quality pathways to certification for aspiring teachers and 
principals. 
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Table B.30. Changes made to state regulations related to teacher 


preparation or certification programs 



Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Reported expanding the types of institutions qualified 
to operate teacher preparation programs to include 
providers operating independently of institutions of 
higher education 




Yes (TL-1 ) 

2011-2012 

4 

2 

5 


2012-2013 

2 

2 

3 


Reported increasing the selectivity of teacher 
preparation programs within the past year 




Yes (TL-1) 

2011-2012 

3 

2 

3 


2012-2013 

2 

1 

8 


Reported increasing the amount of time students in 
teacher preparation programs spend in supervised, 
school-based learning experiences within the past 
year 




Yes (TL-1 ) 

2011-2012 

3 

1 

3 


2012-2013 

3 

0 

7 


Reported increasing the amount of mentoring and/or 
coaching that students in teacher preparation 
programs receive within the past year 




Yes (TL-1) 

2011-2012 

2 

1 

4 


2012-2013 

2 

0 

4 


Reported allowing AC programs for teachers to 
award the same type of certification that traditional 
preparation programs award within the past year 3 




Yes (TL-1 ) 

2011-2012 

4 

2 

4 


2012-2013 

Reported other changes within the past year 

3 

2 

4 

No 

2011-2012 

4 

1 

8 


2012-2013 

3 

1 

10 


Number of States 

10-12 

6-7 

24-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a States that reported that they do not authorize or permit alternative programs for teacher certification are included in 
the analysis of this question as no responses. 

AC = Alternative-route certification; TL-1 = Providing high-quality pathways to certification for aspiring teachers and 
principals. 
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Table B.31. States’ assessment of alternative-route teacher certification 

programs 


Early RTT 

Number of states 

Later RTT 

Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Reported having a process for assessing the 




Yes (TL-2) 

effectiveness of AC programs for teachers in: 

2007-2008® 

9 

5 

14 


2011-2012 

10 

6 

15 


2012-2013 

9 

4 

18 


Reported using the following information to 
assess the effectiveness of AC programs for 
teachers: 

The percentage of enrollees who earn certification 
2011-2012 

5 

3 

9 

No 

2012-2013 

2 

2 

10 


The percentage of enrollees placed in teaching jobs 
2011-2012 

5 

0 

5 

No 

2012-2013 

3 

1 

9 


Rates of retention in the profession 

2011-2012 

4 

0 

4 

No 

2012-2013 

2 

0 

7 


The effectiveness ratings (based in part on student 




Yes (TL-2) 

growth) of credentialed teachers from each program 

2011-2012 

2 

1 

1 


2012-2013 

1 

0 

1 


Qualitative program reviews 

2011-2012 

8 

5 

13 

No 

2012-2013 

7 

4 

14 


Feedback from principals, other school staff, or 




No 

human resources staff on credentialed teachers from 
each program 

2011-2012 

7 

2 

13 


2012-2013 

5 

2 

11 


Other information 

2011-2012 

2 

1 

5 

No 

2012-2013 

4 

1 

8 


Reported using results from assessments of AC 
programs for teachers in the following ways: 

Provided results to certification programs 

2011-2012 

9 

6 

14 

No 

2012-2013 

7 

2 

14 


Provided additional state funding to certification 




Yes (TL-2) 

programs that were shown to be effective 

2011-2012 

1 

0 

0 


2012-2013 

1 

1 

0 


Expanded and/or promoted certification programs 




Yes (TL-2) 

that were shown to be effective 

2011-2012 

3 

0 

1 


2012-2013 

1 

1 

1 


Eliminated and/or reduced state funding for 




No 

certification programs that were shown to be 
ineffective 

2011-2012 

2 

0 

0 


2012-2013 

1 

0 

0 
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Number of states 


Item aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

application 

(subtopic) 

Closed programs that were shown to be ineffective 
2011-2012 

3 

0 

0 

No 

2012-2013 

2 

0 

0 


Publically reported results for each program 
2011-2012 

4 

4 

8 

Yes (TL-2) 

2012-2013 

4 

0 

9 


Other actions 

2011-2012 

2 

1 

2 

No 

2012-2013 

2 

0 

3 



Number of States 9-12 7 25-31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

AC = Alternative-route certification; TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.32. Authorization of alternative-route certification programs for 


principals 

— 

Number of states 

Early RTT Later RTT 

Non-RTT 

Item aligned 
— with RTT 
application 
(subtopic) 

Reported authorizing AC programs for 




Yes 

(TL-2) 

principals in: 






2007-2008 3 

6 

4 

9 



2011-2012 

10 

6 

17 



2012-2013 

10 

6 

15 



Reported authorizing the following groups 






to operate AC programs for principals: 






Institutions of higher education in: 




Yes 

(TL-2) 

2007-2008 3 

6 

4 

8 



2011-2012 

10 

6 

13 



2012-2013 

10 

6 

12 



School districts in: 




Yes 

(TL-2) 

2007-2008 3 

5 

1 

3 



2011-2012 

7 

2 

5 



2012-2013 

7 

3 

7 



Educational service districts or cooperatives in: 




Yes 

(TL-2) 

2007-2008 3 

4 

1 

2 



2011-2012 

5 

2 

4 



2012-2013 

3 

3 

6 



Nonprofit organizations in: 




Yes 

(TL-2) 

2007-2008 3 

3 

3 

2 



2011-2012 

6 

4 

4 



2012-2013 

7 

5 

4 



Partnerships or collaborations among above 




Yes 

(TL-2) 

groups in: 






2007-2008 3 

5 

2 

3 



2011-2012 

9 

3 

8 



2012-2013 

7 

4 

9 



Other groups or organizations in: 




No 


2007-2008 3 

2 

1 

1 



2011-2012 

3 

2 

4 



2012-2013 

2 

3 

4 



Number of States 

9-12 

6-7 

28-30 




Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

AC = Alternative-route certification; TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.33. Changes made to state regulations related to principal 


preparation or certification programs 

Number of states 

Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Reported expanding the types of institutions qualified 
to operate principal preparation programs to include 
providers operating independently of institutions of 
higher education 




Yes (TL-1 ) 

2011-2012 

3 

2 

2 


2012-2013 

2 

3 

1 


Reported increasing the selectivity of principal 
preparation programs within the past year 




Yes (TL-1 ) 

2011-2012 

4 

1 

2 


2012-2013 

3 

2 

5 


Reported increasing the amount of time students in 
principal preparation programs spend in supervised, 
school-based learning experiences within the past 
year 




Yes (TL-1 ) 

2011-2012 

2 

1 

4 


2012-2013 

3 

2 

3 


Reported increasing the amount of mentoring and/or 
coaching that students in principal preparation 
programs receive within the past year 




Yes (TL-1 ) 

2011-2012 

2 

2 

5 


2012-2013 

4 

1 

3 


Reported allowing AC programs for principals to 
award the same type of certification that traditional 
preparation programs award within the past year 3 




Yes (TL-1 ) 

2011-2012 

5 

1 

3 


2012-2013 

2 

1 

2 


Reported other changes within the past year 




No 

2011-2012 

1 

2 

7 


2012-2013 

2 

0 

5 


Number of States 

12 

5-7 

26-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a States that reported that they do not authorize or permit alternative programs for teacher certification are included in 
the analysis of this question as no responses. 

AC = Alternative-route certification; TL-1 = Providing high-quality pathways to certification for aspiring teachers and 
principals. 
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Table B.34. States’ evaluation of alternative-route principal certification 

programs 







Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Reported having a process for assessing the 
effectiveness of AC programs for principals 
in: 

2007-2008 3 

3 

2 

5 

Yes (TL-2) 

2011-2012 

6 

4 

10 


2012-2013 

8 

3 

10 


As of spring 2012/2013, reported using the 
following information to assess the 
effectiveness of AC programs for principals: 

The percentage of enrollees who earn 
certification 




No 

2011-2012 

4 

1 

7 


2012-2013 

4 

3 

5 


The percentage of enrollees placed in school 
administration jobs 




No 

2011-2012 

4 

0 

6 


2012-2013 

Rates of retention in the profession 

2 

0 

4 

No 

2011-2012 

4 

0 

1 


2012-2013 

2 

0 

3 


The effectiveness ratings (based in part on 
student growth) of credentialed principals from 
each program 




Yes (TL-2) 

2011-2012 

2 

0 

1 


2012-2013 

Qualitative program reviews 

2 

0 

0 

No 

2011-2012 

5 

4 

9 


2012-2013 

Feedback from school, district, or human 
resources staff on credentialed principals from 
each program 

7 

3 

7 

No 

2011-2012 

5 

1 

8 


2012-2013 

Other information 

6 

1 

5 

No 

2011-2012 

0 

1 

2 


2012-2013 

1 

1 

1 


Reported using results from assessments of 
AC programs for principals in the following 
ways within the past year: 

Provided results to certification programs 




No 

2011-2012 

4 

3 

9 


2012-2013 

6 

2 

6 


Provided additional state funding to certification 
programs that were shown to be effective 




Yes (TL-2) 

2011-2012 

1 

0 

0 


2012-2013 

2 

0 

0 


Expanded and/or promoted certification 
programs that were shown to be effective 




Yes (TL-2) 

2011-2012 

2 

0 

1 


2012-2013 

1 

0 

0 
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Number of states Item aligned 

with RTT 
application 

Early RTT Later RTT Non-RTT (subtopic) 


Eliminated and/or reduced state funding for No 

certification programs that were shown to be 

ineffective 


2011-2012 

2012-2013 


0 0 0 
0 0 0 


Closed programs that were shown to be 
ineffective 


No 


2011-2012 

2012-2013 


0 0 1 
0 0 1 


Publically reported results for each program 
2011-2012 
2012-2013 
Other actions 
2011-2012 
2012-2013 


1 

3 

0 

1 


3 

1 

1 

0 


5 

4 

2 

0 


No 


No 


Number of States 


11-12 6-7 28-31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

AC = Alternative-route certification; TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.35. State requirements for district adoption of teacher evaluation 
models 


Number of states Item aligned 

with RTT 

application 

Early RTT Later RTT Non-RTT (subtopic) 


Districts must use model prescribed by the state 
2007-2008 3 
2011-2012 
2012-2013 

Districts may adopt state model 
2007-2008 3 
2011-2012 
2012-2013 

Districts must adopt state model if unable to meet 
state expectations 
2007-2008 3 
2011-2012 
2012-2013 

Districts must administer evaluations that comply 
with state statutes and the state monitors the 
evaluation process 
2007-2008 3 
2011-2012 
2012-2013 

Districts must administer evaluations that comply 
with state statutes but the state does not monitor 
the evaluation process 
2007-2008 3 
2011-2012 
2012-2013 


3 

3 

4 

0 

1 

0 


1 

3 

4 


6 

2 

1 


18 

12 

8 


No 


No 


No 


No 


No 


Number of States 


10 


27 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

3 Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 
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Table B.36. Role of student achievement growth in teacher evaluations 




Number of states 

Item aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Reported that student achievement growth was 




Yes (TL-2) 

required 

2007-2008 3 

3 

1 

4 


2011-2012 

7 

5 

16 


2012-2013 

Reported that student achievement growth was 

9 

6 

14 

Yes (TL-2) 

required with a specific weight 

2007-2008 3 

1 

0 

0 


2011-2012 

7 

4 

10 


2012-2013 

Required weight: 

9 

5 

11 

No 

No specific weight required or did not require 
student achievement growth 

2011-2012 

5 

3 

19 


2012-2013 

3 

2 

18 


Between 1 and 20 percent 

2011-2012 

1 

0 

1 


2012-2013 

3 

1 

1 


Between 21 and 50 percent 

2011-2012 

3 

4 

2 


2012-2013 

5 

4 

5 


51 percent or more 

2011-2012 

0 

0 

0 


2012-2013 

0 

0 

0 


“Significant,” “substantial,” or “primary” factor 

2011-2012 

3 

0 

7 


2012-2013 

1 

0 

5 


Number of States 

11-12 

7 

29-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.37. State-reported requirements for performance measures (other 
than student achievement growth) for evaluations of teachers in tested 
grades and subjects 


Number of states 


Item aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Classroom observations 




Yes (TL-2) 

2007-2008 a 

9 

7 

16 


2011-2012 

11 

7 

21 


2012-2013 

11 

7 

21 


Self-assessment 




Yes (TL-2) 

2007-2008 a 

2 

0 

3 


2011-2012 

4 

1 

6 


2012-2013 

2 

0 

7 


Portfolios or other artifacts of teacher practice 




Yes (TL-2) 

2007-2008 a 

3 

1 

3 


2011-2012 

2 

1 

6 


2012-2013 

1 

1 

7 


Peer assessments other than classroom 





observations 




Yes (TL-2) 

2007-2008 a 

0 

1 

0 


2011-2012 

1 

1 

1 


2012-2013 

0 

0 

0 


Student work samples 




Yes (TL-2) 

2007-2008 a 

0 

0 

1 


2011-2012 

2 

1 

3 


2012-2013 

0 

0 

3 


Student surveys or other feedback 




Yes (TL-2) 

2007-2008 a 

0 

0 

1 


2011-2012 

1 

1 

2 


2012-2013 

2 

1 

3 


Parent surveys or other feedback 




Yes (TL-2) 

2007-2008 a 

1 

0 

0 


2011-2012 

2 

1 

3 


2012-2013 

1 

0 

3 


Other measures 




No 

2007-2008 a 

0 

1 

2 


2011-2012 

3 

0 

3 


2012-2013 

3 

1 

2 


Number of States 

11 

7 

26-29 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.38. State-reported requirements for performance measures (other 
than student achievement growth) for evaluations of teachers in nontested 
grades and subjects 

— 

Number of states 

Early RTT Later RTT 

Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Classroom observations 




Yes (TL-2) 

2007-2008 a 

9 

7 

16 


2011-2012 

11 

7 

21 


2012-2013 

11 

7 

22 


Self-assessment 




Yes (TL-2) 

2007-2008 a 

2 

0 

3 


2011-2012 

2 

1 

6 


2012-2013 

3 

0 

7 


Portfolios or other artifacts of teacher practice 




Yes (TL-2) 

2007-2008 3 

3 

1 

3 


2011-2012 

1 

1 

6 


2012-2013 

2 

1 

8 


Peer assessments other than classroom 





observations 




Yes (TL-2) 

2007-2008 a 

0 

1 

0 


2011-2012 

0 

1 

1 


2012-2013 

1 

0 

0 


Student work samples 




Yes (TL-2) 

2007-2008 a 

0 

0 

1 


2011-2012 

1 

1 

3 


2012-2013 

1 

0 

4 


Student surveys or other feedback 




Yes (TL-2) 

2007-2008 3 

0 

0 

1 


2011-2012 

0 

1 

2 


2012-2013 

2 

1 

3 


Parent surveys or other feedback 




Yes (TL-2) 

2007-2008 a 

1 

0 

0 


2011-2012 

1 

1 

3 


2012-2013 

1 

0 

3 


Other measures 




No 

2007-2008 a 

0 

1 

2 


2011-2012 

2 

1 

4 


2012-2013 

4 

1 

3 


Number of States 

11 

7 

29 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.39. Rating levels for overall teacher performance 



Number of states 
(unless otherwise specified) 

Early RTT Later RTT Non-RTT 

Item aligned 
with RTT 
~ application 
(subtopic) 

Reported specifying a required minimum 




Yes (TL-2) 

number of rating levels 

2011-2012 

8 

6 

16 


2012-2013 

Average minimum number of rating levels 3 

9 

6 

15 

No 

2011-2012 

2.7 

2.6 

1.8 


2012-2013 

Number of rating levels states reported 

3.8 

3.4 

1.8 

No 

requiring: 

4 or more levels 

2011-2012 

7 

3 

10 


2012-2013 

8 

6 

11 


3 levels 

2011-2012 

1 

0 

1 


2012-2013 

1 

0 

1 


2 levels 

2011-2012 

0 

3 

5 


2012-2013 

0 

0 

3 


Did not specify a minimum 13 

2011-2012 

2 

1 

13 


2012-2013 

Reported specifying a minimum acceptable 

1 

1 

14 

No 

spread of teachers across rating categories 

2011-2012 

0 

0 

0 


2012-2013 

0 

0 

1 


Number of States 

10-12 

7 

26-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a In the analysis of this question, states that reported not specifying a required minimum number of rating levels were 
counted as zeroes. 

b This includes states that responded no to the question of “Do state regulations specify a required minimum number 
of rating levels to be used when evaluating overall teacher performance? If so, what is the minimum number of rating 
categories that is required?” 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.40. State regulations on tenure and frequency of teacher evaluation 



Number of states 


Item aligned 


(unless otherwise specified) 

with RTT 





application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Allow teachers to earn tenure 3 




No 

2007-2008 b 

8 

6 

27 


2011-2012 

9 

6 

22 


2012-2013 

10 

6 

25 


Have a probationary period for teachers 




No 

2007-2008 b 

11 

7 

29 


2011-2012 

11 

7 

26 


2012-2013 

11 

7 

30 


Of states that reported having a probationary 





period, mean duration of probationary period 





(years) 




No 

2007-2008 b 

2.6 

3.3 

2.8 


2011-2012 

3.3 

3.3 

3.0 


2012-2013 

3.4 

3.3 

2.9 


Evaluated probationary teachers 





At least annually 




Yes (TL-2) 

2007-2008 b 

10 

6 

21 


2011-2012 

10 

6 

23 


2012-2013 

8 

5 

22 


Every other year 




No 

2007-2008 b 

0 

0 

0 


2011-2012 

0 

0 

0 


2012-2013 

0 

0 

0 


Other interval 




No 

2007-2008 b 

0 

1 

4 


2011-2012 

0 

1 

2 


2012-2013 

2 

2 

3 


Evaluated non-probationary teachers 





At least annually 




Yes (TL-2) 

2007-2008 b 

5 

3 

11 


2011-2012 

6 

3 

13 


2012-2013 

8 

3 

13 


Every other year 




No 

2007-2008 b 

2 

0 

1 


2011-2012 

2 

1 

1 


2012-2013 

1 

1 

1 


Other interval 




No 

2007-2008 b 

3 

4 

13 


2011-2012 

2 

3 

11 


2012-2013 

1 

3 

11 


Number of States 

10-12 

7 

24-31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 



Note: A range is provided for the sample sizes because nonresponse varied across items. 


a This category includes states that provide the equivalent of tenure but do not use the term “tenure.” 


b Data from 2007-2008 were collected retrospectively in spring 2012 

. All data provided by states were self-reported 

and not independently verified by the research team. 

For these reasons and potential concerns about recall accuracy, 

readers should exercise caution when interpreting data from 2007-2008. 



TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.41. State regulations on the use of teacher evaluation results 



Number of states 


Item aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

application 

(subtopic) 

Reported requiring results to guide decisions 
about: 

Professional development and support 

2011-2012 

7 

4 

8 

Yes (TL-2) 

2012-2013 

9 

5 

10 


Dismissal 

2011-2012 

7 

6 

8 

Yes (TL-2) 

2012-2013 

7 

5 

7 


Salary increases 

2011-2012 

2 

1 

3 

Yes (TL-2) 

2012-2013 

2 

1 

1 


Bonuses or performance-based compensation 




Yes (TL-2) 

other than salary 

2011-2012 

1 

1 

3 


2012-2013 

3 

1 

1 


Career advancement 

2011-2012 

1 

0 

0 

Yes (TL-2) 

2012-2013 

4 

1 

0 


Other decisions 

2011-2012 

1 

2 

2 

No 

2012-2013 

2 

2 

1 


Reported that evaluation results, rather than 




No 

seniority, are the primary consideration in 
reductions in force and excessing decisions 

2011-2012 

1 

2 

5 


2012-2013 

1 

3 

2 


Reported collecting information on the 




No 

percentage of teachers falling in each rating 
level 

2011-2012 

8 

4 

11 


2012-2013 

8 

5 

9 


Reported analyzing the relationship between 




No 

teacher ratings and student achievement 
growth 

2011-2012 

1 

2 

1 


2012-2013 

4 

4 

0 


Reported providing teacher preparation 




No 

programs with information about 
effectiveness based on: 

Overall teacher ratings 

2011-2012 

1 

0 

1 


2012-2013 

4 

0 

0 


Classroom observations 

2011-2012 

0 

0 

1 


2012-2013 

3 

0 

0 


Student achievement growth 

2011-2012 

3 

0 

0 


2012-2013 

4 

1 

0 


Number of States 

3-12 

3-7 

12-31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


TL-2 = Improving teacher and principal effectiveness based on performance. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.42. State requirements for district adoption of principal evaluation 
models 


Number of states Item aligned 

with RTT 

application 

Early RTT Later RTT Non-RTT (subtopic) 


Districts must use model prescribed by the state 
2007-2008 a 
2011-2012 
2012-2013 

Districts may adopt state model 
2007-2008 a 
2011-2012 
2012-2013 

District must adopt state model If unable to meet 
state expectations 
2007-2008 a 
2011-2012 
2012-2013 

Districts must administer evaluations that comply 
with state statutes and the state monitors the 
evaluation process 
2007-2008 a 
2011-2012 
2012-2013 

Districts must administer evaluations that comply 
with state statutes but the state does not monitor 
the evaluation process 
2007-2008 a 
2011-2012 
2012-2013 


4 

4 

1 


4 

5 

6 

3 

3 

3 


1 

4 

6 


17 

12 

9 


No 


No 


No 


No 


No 


Number of States 


25 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.43. Role of student achievement growth in principal evaluations 



Early RTT 

Number of states 

Later RTT 

Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Reported that student achievement growth 




Yes (TL-2) 

was required 

2007-2008 3 

2 

1 

2 


2011-2012 

5 

2 

11 


2012-2013 

Reported that student achievement growth 

7 

4 

12 

Yes (TL-2) 

was required with a specific weight 

2007-2008 3 

2 

0 

0 


2011-2012 

7 

2 

5 


2012-2013 

Required weight: 

8 

5 

9 

No 

No specific weight required or did not require 
student achievement growth 

2011-2012 

5 

5 

24 


2012-2013 

4 

2 

20 


Between 1 and 20 percent 

2011-2012 

1 

0 

0 


2012-2013 

3 

2 

2 


Between 21 and 50 percent 

2011-2012 

0 

0 

0 


2012-2013 

4 

3 

4 


51 percent or more 

2011-2012 

0 

0 

0 


2012-2013 

0 

0 

0 


“Significant,” “substantial,” or “primary” factor 

2011-2012 

3 

0 

3 


2012-2013 

1 

0 

3 


Number of States 

9-12 

5-7 

28-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.44. State-reported requirements for performance measures for 
principal evaluations (other than student achievement growth) 


Number of states Item aligned 

with RTT 

application 

Early RTT Later RTT Non-RTT (subtopic) 


Self-assessment 

2007-2008 3 

2011-2012 

2012-2013 

District administrator input 
2007-2008 3 
2011-2012 
2012-2013 

Staff surveys or other feedback 
2007-2008 a 
2011-2012 
2012-2013 

Student surveys or other feedback 
2007-2008 3 
2011-2012 
2012-2013 

Other measures 
2007-2008 a 
2011-2012 
2012-2013 


5 

7 

7 

10 

12 

16 

3 

5 

5 

1 

3 

3 

4 
9 
7 


Yes (TL-2) 


Yes (TL-2) 


Yes (TL-2) 


Yes (TL-2) 


No 


Number of States 


10 


28 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.45. Rating levels for overall principal performance 


Number of states Item aligned 

(unless otherwise specified) with RTT 

application 

Early RTT Later RTT Non-RTT (subtopic 


Reported specifying a required minimum Yes (TL-2) 

number of rating levels 


2007-2008 3 

5 

2 

6 


2011-2012 

7 

3 

14 


2012-2013 

Average minimum number of rating levels b 

9 

4 

13 

No 

2007-2008 3 

1.4 

0.6 

0.6 


2011-2012 

2.7 

2.3 

1.6 


2012-2013 

Number of rating levels states reported 

3.8 

3.4 

1.5 

No 

requiring: 

4 or more levels 

2011-2012 

7 

3 

9 


2012-2013 

10 

6 

9 


3 levels 

2011-2012 

0 

0 

3 


2012-2013 

1 

0 

1 


2 levels 

2011-2012 

1 

2 

2 


2012-2013 

0 

0 

3 


Did not specify a minimum 3 

2011-2012 

4 

2 

15 


2012-2013 

1 

1 

16 


Number of States 

10-12 

5-7 

29-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

b In the analysis of this question, states that reported not specifying a required minimum number of rating levels were 
counted as zeroes. 

c This includes states that responded no to the question of “Do state regulations specify a required minimum number 
of rating levels to be used when evaluating overall principal performance? If so, what is the minimum number of rating 
categories that is required?” 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.46. State uses of principal evaluation results 




Number of states 


Item aligned 
with RTT 


Early RTT Later RTT 

Non-RTT 

application 

(subtopic) 

Reported requiring results to guide decisions 
about: 

Professional development and support 

2011-2012 

4 

3 

3 

Yes (TL-2) 

2012-2013 

4 

4 

4 


Dismissal 

2011-2012 

4 

3 

5 

Yes (TL-2) 

2012-2013 

5 

4 

3 


Salary increases 

2011-2012 

1 

1 

1 

Yes (TL-2) 

2012-2013 

1 

1 

0 


Bonuses or performance-based compensation other 
than salary 

2011-2012 

1 

0 

1 

Yes (TL-2) 

2012-2013 

1 

1 

1 


Career advancement 

2011-2012 

1 

0 

0 

Yes (TL-2) 

2012-2013 

1 

1 

0 


Other decisions 

2011-2012 

0 

1 

1 

No 

2012-2013 

1 

3 

1 

No 

Reported collecting information on the 
percentage of principals falling in each rating 
level 

2011-2012 

6 

5 

7 

2012-2013 

7 

4 

7 

No 

Reported analyzing the relationship between 
principal ratings and student achievement growth 

2011-2012 

1 

0 

0 

2012-2013 

3 

3 

2 


Reported providing principal preparation 
programs with information about effectiveness 
based on: 

Overall principal ratings 

2011-2012 

1 

0 

1 

No 

2012-2013 

1 

0 

0 


Student achievement growth 

2011-2012 

0 

0 

0 

No 

2012-2013 

1 

0 

0 


Something else 

2011-2012 

0 

0 

1 

No 

2012-2013 

0 

0 

1 


Number of States 

5-12 

5-7 

22-30 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


TL-2 = Improving teacher and principal effectiveness based on performance. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.47. Actions taken by states to identify areas of teacher shortages 




Number of states 

Item aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Reported having a process to identify areas 
of teacher shortage 

2011-2012 

11 

7 

29 

Yes (TL-3) 

2012-2013 

12 

6 

31 


Reported taking actions to identify areas of 
teacher shortage: 

Reviewed the number of vacant positions and/or 
teacher attrition by content area, grade, student 
achievement levels, or student demographic 
characteristics 

2011-2012 

9 

4 

20 

No 

2012-2013 

10 

5 

25 


Reviewed the number of positions filled by 
teachers teaching in subjects outside their 
certification area 

2011-2012 

11 

4 

25 

No 

2012-2013 

11 

5 

26 


Reviewed the projected total number of teacher 
preparation program graduates 

2011-2012 

8 

0 

16 

No 

2012-2013 

7 

2 

21 


Reviewed the projected number of teacher 
preparation program graduates in specific 
subject areas 

2011-2012 

8 

1 

14 

No 

2012-2013 

7 

1 

19 


Reviewed the projected number of teacher 
preparation program graduates in specific grade 
levels 

2011-2012 

6 

1 

11 

No 

2012-2013 

5 

0 

13 


Surveyed district administrators 

2011-2012 

3 

1 

20 

No 

2012-2013 

5 

1 

20 


Surveyed school administrators 

2011-2012 

1 

1 

8 

No 

2012-2013 

2 

0 

10 


Other actions 

2011-2012 

5 

2 

8 

No 

2012-2013 

6 

1 

6 


Number of States 

11-12 

7 

29-31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.48. Full-time equivalent staff and actions taken by states to address 
current areas of teacher shortages 


Number of states 

Item aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Mean number of full-time equivalent staff 3 

151,958 

151,862 

84,265 

No 

Reported having a teacher shortage 




No 

2011-2012 

10 

7 

28 


2012-2013 

11 

6 

28 


Reported taking actions to address teacher 





shortage: 





Enacted policies allowing traditional teacher 




Yes (TL-3) 

preparation programs to expand the number of 





students served 





2011-2012 

1 

0 

1 


2012-2013 

1 

0 

0 


Enacted policies allowing AC programs for 




Yes (TL-3) 

teachers to expand the number of students 





served 





2011-2012 

3 

1 

2 


2012-2013 

2 

0 

4 


Provided incentives for teachers to become 




Yes (TL-3) 

certified in areas of shortage 





2011-2012 

6 

2 

10 


2012-2013 

6 

2 

11 


Notified teacher preparation programs about 




Yes (TL-3) 

the areas of shortage 





2011-2012 

9 

5 

19 


2012-2013 

10 

4 

22 


Encouraged teacher preparation programs to 




Yes (TL-3) 

train more teachers in the areas of shortage 





2011-2012 

8 

4 

22 


2012-2013 

10 

2 

22 


Developed new AC programs for teachers 




Yes (TL-3) 

focused on the areas of shortage 





2011-2012 

3 

4 

8 


2012-2013 

5 

2 

7 


Increased the amount of induction support for 




Yes (TL-3) 

novice teachers with the goal of reducing 





teacher attrition in areas of shortage 





2011-2012 

6 

0 

7 


2012-2013 

7 

0 

8 


Other actions 




No 

2011-2012 

1 

0 

9 


2012-2013 

6 

0 

5 


Number of States 

6-11 

4-7 

16-29 



Source: Interviews with state administrators in spring 2012 and spring 2013, Common Core of Data. 
Note: A range is provided for the sample sizes because nonresponse varied across items. 


a Full-time equivalent (FTE) is the amount of time required to perform an assignment stated as a proportion of a full- 
time position. It is computed by dividing the amount of time employed by the amount of time normally required for a 
full-time position. For example, two half-time employees represent one FTE. 

AC = Alternative-route certification, FTE = full-time equivalent; TL-3 = Ensuring equitable distribution of effective 
teachers and principals. 
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Table B.49. State analyses of shifts in 

teacher distribution 



- 

Number of states 

Early RTT Later RTT 

Non-RTT 

Item aligned 
with RTT 
application 
(subtopic) 

Reported conducting analyses to determine 
whether the distribution of effective teachers 
between high-poverty or high-minority schools 
and low-poverty or low-minority schools shifted 

2011-2012 

4 

2 

8 

Yes (TL-3) 

2012-2013 

5 

2 

6 


Reported examining factors in the analysis of 
teacher distribution: 

Teacher effectiveness based on student achievement 
growth 

2011-2012 

1 

0 

2 

Yes (TL-3) 

2012-2013 

4 

1 

0 


Proportions of novice and experienced teachers 
2011-2012 

2 

2 

6 

No 

2012-2013 

3 

2 

5 


Proportion of certified teachers 

2011-2012 

4 

1 

7 

No 

2012-2013 

3 

2 

6 


Proportion of teachers assigned to grades or classes 
outside of their field of certification 

2011-2012 

4 

1 

7 

No 

2012-2013 

2 

2 

6 


Other factors 

2011-2012 

1 

2 

5 

No 

2012-2013 

2 

1 

0 


Reported that analyses of distribution of effective 
teachers showed the following pattern: 

A more equitable distribution 

2011-2012 

1 

0 

3 

No 

2012-2013 

1 

0 

1 


A less equitable distribution 

2011-2012 

0 

1 

1 

No 

2012-2013 

0 

0 

0 


No change in the distribution 

2011-2012 

1 

0 

4 

No 

2012-2013 

1 

1 

5 


Reported sending results from analyses of 
teacher distribution to the districts included in 
such analyses 

2011-2012 

4 

2 

7 

No 

2012-2013 

3 

2 

5 


Number of States 

8-12 

6-7 

28-29 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.50. State strategies for promoting equitable distribution of effective 

teachers 







Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Reported offering more compensation for 
teachers who work in high-poverty or high- 
minority schools 




Yes (TL-3) 

2011-2012 

3 

1 

3 


2012-2013 

4 

0 

2 


Reported offering performance-based 
compensation for effective teachers who work 
in high-poverty or high-minority schools 




Yes (TL-3) 

2011-2012 

2 

0 

1 


2012-2013 

1 

0 

1 


Reported earlier hiring of staff to work in high- 
poverty or high-minority schools and districts 




Yes (TL-3) 

2011-2012 

3 

0 

0 


2012-2013 

2 

0 

2 


Reported increasing recruitment activities 




Yes (TL-3) 

2011-2012 

4 

1 

3 


2012-2013 

2 

0 

3 


Reported improving teaching and learning 
environments at high-poverty or high-minority 
schools 




Yes (TL-3) 

2011-2012 

2 

1 

4 


2012-2013 

2 

0 

8 


Reported offering more professional 
development, mentoring, or induction for 
teachers working in high-poverty or high- 
minority schools 




Yes (TL-3) 

2011-2012 

4 

0 

4 


2012-2013 

3 

0 

8 


Reported other strategies 




No 

2011-2012 

1 

2 

6 


2012-2013 

2 

1 

4 


Number of States 

12 

7 

29-30 


Source: Interviews with state administrators in spring 2012 and spring 2013. 



Note: A range is provided for the sample sizes because nonresponse varied across items. 


TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.51. Actions taken by states to identify areas of principal shortages 


- 

Early RTT 

Number of states 

Later RTT 

Non-RTT 

Item aligned 

J with RTT 
application 
(subtopic) 

Reported having a process to identify areas of 
principal shortages 

2011-2012 

5 

0 

7 

Yes (TL-3) 

2012-2013 

4 

2 

13 


Reported taking actions to identify areas of 
principal shortages: 

Reviewed the number of vacant positions and/or 
principal attrition by student achievement levels or 
student demographic characteristics 

2011-2012 

3 

0 

4 

No 

2012-2013 

1 

0 

9 


Reviewed the projected total number of principal 
preparation program graduates 

2011-2012 

3 

0 

6 

No 

2012-2013 

2 

1 

7 


Surveyed district administrators 

2011-2012 

3 

0 

4 

No 

2012-2013 

2 

0 

4 


Surveyed school administrators 

2011-2012 

1 

0 

2 

No 

2012-2013 

0 

0 

2 


Other actions 

2011-2012 

1 

0 

2 

No 

2012-2013 

1 

1 

2 


Number of States 

10-11 

7 

30-31 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.52. Actions taken by states to address current areas of principal 

shortages 







Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Reported having a principal shortage 




No 

2011-2012 

5 

0 

7 


2012-2013 

4 

0 

10 


Reported taking actions to address principal 
shortage: 

Enacted policies allowing traditional principal 
preparation programs to expand the number of 
students served 




Yes (TL-3) 

2011-2012 

1 

0 

1 


2012-2013 

Enacted policies allowing AC programs for 
principals to expand the number of students 
served 

1 

0 

0 

Yes (TL-3) 

2011-2012 

2 

0 

2 


2012-2013 

Provided incentives for individuals to become 
certified to work as principals 

2 

0 

0 

Yes (TL-3) 

2011-2012 

2 

0 

1 


2012-2013 

2 

0 

0 


Notified principal preparation programs about the 
areas of shortage 




Yes (TL-3) 

2011-2012 

3 

0 

3 


2012-2013 

2 

0 

4 


Encouraged principal preparation programs to 
train more principals in the areas of shortage 




Yes (TL-3) 

2011-2012 

3 

0 

2 


2012-2013 

2 

0 

4 


Developed new AC programs for principals 
focused on the areas of shortage 




Yes (TL-3) 

2011-2012 

4 

0 

2 


2012-2013 

Increased the amount of support for principals 
with the goal of reducing principal attrition in areas 
of shortage 

1 

0 

0 

Yes (TL-3) 

2011-2012 

4 

0 

1 


2012-2013 

Other actions 

1 

0 

5 

No 

2011-2012 

0 

0 

1 


2012-2013 

1 

0 

3 


Number of States 

8-10 

3 

18-25 


Source: Interviews with state administrators in spring 2012 and spring 2013. 



Note: A range is provided for the sample sizes because nonresponse varied across items. 


AC = Alternative-route certification; TL-3 = Ensuring equitable distribution of effective teachers and principals. 


B-58 




APPENDIX B RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table B.53. State analyses of shifts 

in principal distribution 





Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT Non-RTT 

(subtopic) 

Reported conducting analyses to determine 
whether the distribution of effective 




Yes (TL-3) 

principals between high-poverty or high- 
minority schools and low-poverty or low- 
minority schools shifted 





2011-2012 

1 

0 

1 


2012-2013 

1 

1 

0 


Reported examining factors in the analysis of 
principal distribution: 

Principal experience 




Yes (TL-3) 

2011-2012 

1 

0 

1 


2012-2013 

Principal effectiveness ratings based on student 
achievement growth 

1 

1 

0 

No 

2011-2012 

1 

0 

0 


2012-2013 

Other factors 

1 

1 

0 

No 

2011-2012 

0 

0 

1 


2012-2013 

1 

0 

0 


Reported that analyses of distribution of 
effective principals showed the following 
pattern: 

A more equitable distribution 




No 

2011-2012 

1 

0 

0 


2012-2013 

A less equitable distribution 

0 

0 

0 

No 

2011-2012 

0 

0 

0 


2012-2013 

No change in the distribution 

0 

0 

0 

No 

2011-2012 

0 

0 

1 


2012-2013 

0 

1 

0 


Reported sending results from analyses of 
principal distribution to the districts included 
in such analyses 




No 

2011-2012 

1 

0 

1 


2012-2013 

1 

0 

0 


Number of States 

11-12 

7 

29 


Source: Interviews with state administrators in spring 2012 and spring 2013. 



Note: A range is provided for the sample sizes because nonresponse varied across items. 


TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.54. State strategies for promoting equitable distribution of effective 

principals 







Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Reported offering more compensation for principals 
who work at high-poverty or high-minority schools 




Yes (TL-3) 

2011-2012 

2 

0 

0 


2012-2013 

2 

0 

1 


Reported offering performance-based 
compensation for effective principals of high- 
poverty or high-minority schools 




Yes (TL-3) 

2011-2012 

2 

0 

0 


2012-2013 

1 

0 

0 


Reported earlier hiring of staff in high-poverty or 
high-minority schools and districts 




Yes (TL-3) 

2011-2012 

2 

0 

0 


2012-2013 

0 

0 

2 


Reported increasing external recruitment activities 




Yes (TL-3) 

2011-2012 

1 

1 

1 


2012-2013 

0 

0 

3 


Reported improving teaching and learning 
environments at high-poverty or high-minority 
schools 




Yes (TL-3) 

2011-2012 

2 

1 

3 


2012-2013 

2 

1 

6 


Reported offering more professional development 
for principals working at high-poverty or high- 
minority schools 




Yes (TL-3) 

2011-2012 

3 

0 

1 


2012-2013 

2 

1 

6 


Reported other strategies 




No 

2011-2012 

1 

0 

2 


2012-2013 

0 

0 

3 


Number of States 

11-12 

7 

29-30 


Source: Interviews with state administrators in spring 2012 and spring 2013. 



Note: A range is provided for the sample sizes because nonresponse varied across items. 


TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.55. Organizational structures intended to improve state capacity to 


support school turnaround 




Number of states 


Item aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Reported having organizational or 




Yes (TA-2) 

administrative structures intended to improve 
state capacity to support school turnaround in: 

2007-2008 3 

7 

6 

12 


2011-2012 

12 

7 

26 


2012-2013 

12 

7 

26 


Reported having the following organizational or 
administrative structures: 

State office explicitly designated to support school 




No 

turnaround in: 

2007-2008 3 

3 

3 

4 


2011-2012 

12 

5 

11 


2012-2013 

Regional offices explicitly designated to support 

11 

5 

16 

No 

school turnaround in: 

2007-2008 3 

1 

0 

1 


2011-2012 

3 

2 

3 


2012-2013 

Contracts with external consultants to support 

3 

2 

4 

No 

school turnaround in: 

2007-2008 3 

5 

5 

8 


2011-2012 

8 

5 

16 


2012-2013 

State Department of Education staff explicitly 

9 

3 

20 

No 

designated to support school turnaround, but no 
state-level turnaround office in: 

2007-2008 3 

3 

2 

5 


2011-2012 

1 

1 

15 


2012-2013 

Regional staff explicitly designated to support 

3 

2 

12 

No 

school turnaround, but no regional state turnaround 
offices, in: 

2007-2008 3 

1 

1 

1 


2011-2012 

2 

3 

6 


2012-2013 

State-level staff or consultants to provide support to 

2 

2 

8 

No 

turnaround schools and districts in working with 

ELLs in: 

2007-2008 3 

3 

2 

6 


2011-2012 

6 

4 

13 


2012-2013 

Monitoring or reporting requirements specifically for 

8 

3 

13 

No 

schools receiving SIG and/or RTT funds to 
implement a school intervention model in: 

2007-2008 3 

3 

3 

6 


2011-2012 

11 

5 

26 


2012-2013 

Other structures reported in: 

11 

5 

26 

No 

2007-2008 3 

1 

2 

1 


2011-2012 

4 

5 

5 


2012-2013 

4 

2 

5 


Number of States 

9-12 

5-7 

25-30 


Source: Interviews with state administrators in spring 2012 and spring 2013. 
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Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

TA-2 = Turning around the lowest-achieving schools; ELLs = English language learners; SIG = School Improvement 
Grant. 
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Table B.56. Flexibility granted by states to persistently lowest-achieving 

schools 







Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non -RTT 

(subtopic) 

Reported providing their persistently lowest- 
achieving schools with flexibility from 
collective bargaining agreements or state 
policies that guide staffing in: 




Yes (TA-2) 

2007-2008 a 

1 

1 

1 


2011-2012 

4 

1 

3 


2012-2013 

2 

0 

3 


Reported providing flexibility on the following 
aspects of collective bargaining agreements or 
state policies: 

Procedures for assigning or removing staff in: 




No 

2007-2008 a 

1 

1 

0 


2011-2012 

3 

1 

1 


2012-2013 

2 

0 

2 


Requirements or policies related to staff hours and 
responsibilities in: 




No 

2007-2008 3 

1 

1 

0 


2011-2012 

3 

0 

2 


2012-2013 

Procedures related to the distribution of effective 
staff in: 

2 

0 

3 

No 

2007-2008 a 

1 

0 

0 


2011-2012 

2 

0 

1 


2012-2013 

Other flexibility in: 

1 

0 

2 

No 

2007-2008 3 

1 

1 

1 


2011-2012 

2 

1 

2 


2012-2013 

1 

0 

1 


Number of States 

7-8 

6 

24 


Source: Interviews with state administrators in spring 2012 and spring 2013. 



Note: A range is provided for the sample sizes because nonresponse varied across items. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 

and not independently verified by the research team. 

For these reasons and potential concerns about recall accuracy, 

readers should exercise caution when interpreting data from 2007- 

-2008. 



TA-2 = Turning around the lowest-achieving schools. 
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Table B.57. State teacher assignment laws and policies for persistently 


lowest-achieving schools 



Number of states 


Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

Reported having laws, regulations, or 
policies that specifically address teacher 
assignment for their persistently lowest- 
achieving schools in: 




Yes (TA-2) 

2007-2008 3 

3 

0 

0 


2011-2012 

4 

2 

0 


2012-2013 

7 

1 

0 


Reported that their teacher assignment laws 
or policies for their persistently lowest- 
achieving schools have the following 
features: 





Financial and/or other incentives for teachers to 
begin or continue to work in the state’s lowest- 
achieving schools in: 




Yes (TA-2) 

2007-2008 3 

2 

0 

0 


2011-2012 

4 

2 

0 


2012-2013 

Financial and/or other incentives for staff with 

ELL expertise to begin or continue to work in 
the state’s lowest-achieving schools in: 

6 

1 

0 

No 

2007-2008 3 

0 

0 

0 


2011-2012 

0 

0 

0 


2012-2013 

2 

0 

0 


School discretion or authority to decide which 
staff to hire for the lowest-achieving schools in: 




Yes (TA-2) 

2007-2008 3 

1 

0 

0 


2011-2012 

4 

2 

0 


2012-2013 

6 

1 

0 


Teacher tenure rules that affect placement in or 
removal from the lowest-achieving schools in: 




Yes (TA-2) 

2007-2008 3 

2 

0 

0 


2011-2012 

2 

0 

0 


2012-2013 

Other teacher assignment policies in: 

3 

0 

0 

No 

2007-2008 3 

0 

0 

0 


2011-2012 

0 

0 

0 


2012-2013 

1 

0 

0 


Number of States 

11-12 

7 

29 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

TA-2 = Turning around the lowest-achieving schools; ELL = English language learner. 
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Table B.58. State authority to take over failing schools 




Number of states 

Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Reported having authority to take over failing 
schools in: 

2007-2008 a 

3 

4 

9 

Yes (TA-1 ) 

2011-2012 

5 

5 

11 


2012-2013 

3 

4 

10 


Number of States 

10 

7 

31 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

TA-1 = Authority to intervene in the lowest-achieving schools and LEAs. 
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Table B.59. State restrictions on 

new charter authorization and charter 

enrollment 






Number of states (unless otherwise specified) 

Item aligned 





with RTT 
application 


Early RTT 

Later RTT 

Non-RTT 

(subtopic) 

As of spring 2012, number of states that 
currently permit or have ever permitted the 
authorization of charter schools 




Yes (CH-1) 

2011-2012 

12 

6 

23 


2012-2013 

12 

6 

25 


Among states that currently permit or have 
ever permitted the authorization of charter 
schools, percentage of state public schools 
that are charter schools in: 




No 

2007-2008 3 

7.2 

7.0 

3.7 


2011-2012 

9.6 

8.7 

4.5 


2012-2013 

10.1 

9.1 

4.9 


Among states that currently permit or have 
ever permitted the authorization of charter 
schools, percentage of state public students 
that attend charter schools in: 




No 

2007-2008 3 

5.1 

4.3 

2.2 


2011-2012 

7.1 

6.5 

3.5 


2012-2013 

7.6 

7.0 

3.8 


Among states that currently permit or have 
ever permitted the authorization of charter 
schools, number that reported having 
restrictions on the creation of new charter 
schools or charter enrollment in: 




Yes (CH-1) 

2007-2008 3 

8 

2 

15 


2011-2012 

6 

1 

10 


2012-2013 

6 

2 

10 


Among states that currently permit or have 
ever permitted the authorization of charter 
schools, number that reported the following 
restrictions on the creation of new charter 
schools or charter enrollment: 

Caps on total or maximum number of charter 
schools operating overall in: 




No 

2007-2008 3 

5 

2 

7 


2011-2012 

3 

1 

2 


2012-2013 

Caps on the number of new charters schools 
that may be authorized in: 

4 

2 

3 

No 

2007-2008 3 

2 

0 

5 


2011-2012 

2 

0 

2 


2012-2013 

3 

1 

3 


Restrictions on the number or percentage of 
students who may enroll in charter schools in: 




No 

2007-2008 3 

1 

0 

1 


2011-2012 

1 

1 

2 


2012-2013 

2 

1 

3 
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Number of states (unless otherwise specified) 

Item aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
application 
(subtopic) 

Charter schools restricted to specific districts or 




No 

localities in: 





2007-2008 3 

1 

1 

2 


2011-2012 

3 

1 

2 


2012-2013 

3 

1 

4 


Moratorium on new charter schools in: 




No 

2007-2008 3 

2 

0 

3 


2011-2012 

1 

0 

0 


2012-2013 

0 

0 

0 


Other restrictions reported in: 




No 

2007-2008 3 

3 

1 

2 


2011-2012 

1 

1 

2 


2012-2013 

0 

1 

2 


Number of States 

12 

6-7 

20-31 



Source: Interviews with state administrators in spring 2012 and spring 2013; National Alliance for Public Charter 
Schools (2013). 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

CH-1 = Eliminating restrictions on charter school creation and enrollment. 
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Table B.60. Charter school authorizers 



Among states that currently permit or have ever 
permitted the authorization of charter schools, 
number of states 

Item aligned 


Early RTT 

Later RTT 

Non-RTT 

With K 1 1 
application 
(subtopic) 

Reported the following entities are 
permitted by state law to authorize 
charter schools: 

State department of education and/or state 
board of education in: 

2007-2008 a 

7 

5 

13 

No 

2011-2012 

9 

4 

12 


2012-2013 

10 

6 

14 


Districts in: 

2007-2008 a 

8 

5 

18 

No 

2011-2012 

8 

5 

18 


2012-2013 

7 

5 

18 


Regional educational authority in: 

2007-2008 3 

2 

0 

2 

No 

2011-2012 

1 

1 

2 


2012-2013 

1 

1 

3 


Independent charter school board in: 
2007-2008 3 

3 

2 

3 

No 

2011-2012 

2 

2 

5 


2012-2013 

2 

1 

3 


Municipal government in: 

2007-2008 3 

3 

0 

2 

No 

2011-2012 

1 

1 

3 


2012-2013 

1 

1 

2 


Colleges and/or universities in: 

2007-2008 3 

4 

1 

6 

No 

2011-2012 

3 

2 

10 


2012-2013 

4 

2 

10 


Other non-profit organizations in: 

2007-2008 3 

1 

0 

1 

No 

2011-2012 

1 

1 

1 


2012-2013 

1 

1 

2 


Number of States 

12 

6 

18-22 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 
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Table B.61. State monitoring of charter schools 


Among states that currently permit or have 
ever permitted the authorization of charter 

schools, number of states Item aligned 

with RTT 

application 

Early RTT Later RTT Non-RTT (subtopic) 


Reported having mechanisms in place to 
monitor the performance of charter schools in: 


2007-2008 a 1 1 

2011-2012 12 

2012-2013 12 

Reported monitoring charter school 
performance on the following dimensions: 

Academic performance in: 

2007-2008 a 1 1 

2011-2012 12 

2012-2013 12 

Enrollment in: 

2007-2008 3 9 

2011-2012 10 

2012-2013 11 

Proportion of different student populations served 
in: 

2007-2008 a 8 

2011-2012 10 

2012-2013 6 

Finances and budget in: 

2007-2008 3 1 1 

2011-2012 12 

2012-2013 11 

Facilities in: 

2007-2008 3 10 

2011-2012 11 

2012-2013 9 


Yes (CH-2) 


6 20 
6 20 
6 20 


6 

6 

6 

4 

4 

6 


Yes (CH-2) 

19 

20 
20 

Yes (CH-2) 
15 
15 
13 

Yes (CH-2) 


4 

4 

5 

6 
6 
6 

4 

4 

2 


9 

9 

8 

Yes (CH-2) 

18 

19 

19 

Yes (CH-2) 
13 
12 
12 


Number of States 


12 6 21 


Source: Interviews with state administrators in spring 2012 and spring 2013. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

CH-2 = Refining authorization and monitoring processes for charter schools. 
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Table B.62. Charter school applications and authorizations 



Among states that currently permit or have 
ever permitted the authorization of charter 
schools, mean number (unless otherwise 
specified) 

Item aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

application 

(subtopic) 

Mean number of charter schools operating in the 

137.9 

168.7 

98.8 

No 

state in the 201 1-2012 school year 





Mean number of charter schools operating in the 

155.2 

185.8 

110.8 

No 

state in the 2012-2013 school year 





Mean number of new charter school applications 




No 

submitted in the state in: 





2007-2008 a 

21.6 

24.4 

8.2 


2011-2012 

49.4 

30.8 

7.8 


2012-2013 

57.9 

29.8 

7.1 


Mean number of new charter schools authorized 




No 

in the state in: 





2007-2008 3 

11.5 

10.0 

11.6 


2011-2012 

16.1 

16.5 

11.0 


2012-2013 

19.8 

9.2 

12.4 


Mean percentage of yearly charter school 




No 

applications that were authorized in: 





2007-2008 3 

34.0 

47.5 

52.8 


2011-2012 

37.6 

63.6 

42.0 


2012-2013 

31.7 

33.7 

47.4 


Number of States 

9-12 

4-6 

9-23 



Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 
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Table B.63. Charter school closures 


Among states that currently permit or have 
ever permitted the authorization of charter 
schools, mean number (unless otherwise 
specified) 


Early RTT Later RTT 


Non-RTT 


Item 
aligned 
with RTT 
application 
(subtopic) 


Mean number of charter schools voluntarily closed No 

in the state in: 


2007-2008 a 

1.5 

4.8 

0.4 


2011-2012 

1.3 

5.4 

1.3 


2012-2013 

Mean number of charter schools whose charters 

1.8 

1.0 

0.6 

No 

were revoked or not renewed in: 

2007-2008 3 

0.5 

0.4 

1.2 


2011-2012 

1.1 

1.8 

1.9 


2012-2013 

Mean number of charter schools whose charters 

1.2 

1.8 

1.5 

No 

were revoked or not renewed for academic 
reasons in: 

2007-2008 3 

0.1 

0.3 

0.1 


2011-2012 

0.7 

2.3 

0.6 


2012-2013 

Mean percentage of (involuntary) charter school 

0.6 

2.3 

0.8 

No 

closures that were due to academic reasons in: 

2007-2008 3 

0.0 

100.0 

10.0 


2011-2012 

75.0 

100.0 

10.7 


2012-2013 

50.0 

100.0 

25.0 


Number of States 

2-12 

1-6 

2-23 



Source: Interviews with state administrators in spring 2012 and spring 2013. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 
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This appendix provides a crosswalk between each policy or practice aligned with the RTT 
application criteria and the 2013 state administrator interview questions. For each of the six RTT 
topic areas addressed in Chapter IV, this appendix presents a table showing the 2013 interview 
questions that address the policies and practices aligned with the RTT application criteria (Tables 
C.l through C.6). This appendix also presents a table showing the 2013 interview questions that 
addressed the ELL-focused policies and practices aligned with the RTT application criteria 
(Table C.7). Mathematica Policy Research (2014) contains the 2013 state interview protocol. 


Table C.l. Interview questions addressing the state capacity (SC) topic area 
policies and practices 


Policy or practice Interview questions addressing the state capacity policy or practice 


RTT application criterion 

• The state has set forth a comprehensive and coherent reform agenda that clearly articulates its 
goals for implementing reforms in the four education areas described in the ARRA and improving 
student outcomes statewide, establishes a clear and credible path to achieving these goals, and is 
consistent with the specific reform plans that the state proposed throughout its application 

Having a comprehensive SCI . Does your state have a comprehensive education reform plan in place, 

education reform plan in place which means a plan that is intended to result in state-wide improvements in 

student achievement? 


Prioritizing the adoption and 
implementation of college and 
career ready standards or 
assessments 


SC3.To what extent (great extent, moderate extent, little extent, or not at all) is 
each of the following educational reform initiatives a priority in your state for the 
current school year? a. Adopting and implementing college and career ready 
standards; b. Adopting and implementing college and career ready 
assessments 


Prioritizing the development of 
comprehensive, student-level, 
longitudinal data systems or 
using data to improve 
instruction 


SC3.To what extent (great extent, moderate extent, little extent, or not at all) is 
each of the following educational reform initiatives a priority in your state for the 
current school year? c. Building comprehensive, student-level, longitudinal data 
systems; d. Using data to improve instruction 


Prioritizing the recruiting, 
developing, rewarding, and 
retaining of effective teachers 
and principals, especially 
where they are needed most 
(including developing and 
implementing evaluation 
systems) 


SC3.To what extent (great extent, moderate extent, little extent, or not at all) is 
each of the following educational reform initiatives a priority in your state for the 
current school year? e. Recruiting effective teachers and school leaders; f. 
Retaining effective teachers and school leaders; g. Rewarding effective 
teachers and school leaders, for example, by offering bonuses and/or increased 
compensation to effective teachers; h. Developing and preparing effective 
teachers and school leaders; i. Developing and implementing a teacher and 
principal evaluation system that is based on student growth; j. Improving the 
distribution of effective teachers and principals 


Prioritizing the turnaround of its SC3.To what extent (great extent, moderate extent, little extent, or not at all) is 
lowest-achieving schools each of the following educational reform initiatives a priority in your state for the 

current school year? k. Turning around the lowest-achieving schools 

RTT application criteria 

• Ensure state has the capacity required to implement its proposed plans by providing strong 
leadership and dedicated teams to implement the statewide education reform plans; supporting 
participating LEAs in successfully implementing the education reform plans the state has 
proposed 

• Use support from a broad group of stakeholders to better implement its plans, as evidenced by 
the strength of the statements or actions of support from the state’s teachers and principals and 
other critical stakeholders 
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Policy or practice Interview questions addressing the state capacity policy or practice 


Providing leadership or teams 
to support the implementation 
of reform plans 

SC6. To what extent does the state education agency play each of the following 
roles (a great extent, moderate extent, little extent, or not at all)? a. Creator of a 
state-wide vision for reforms such as articulating a vision and direction for 
educational improvement in the state; d. Facilitator between educational units 
and external expertise; e. Supporter providing direct support services to districts 
and schools? 

SC7. Within the past year, has the state education agency provided targeted 
support to certain types of districts or schools for implementation of statewide 
education reform priorities? By targeted supports, we mean supports that are 
provided to a specific group of schools and/or districts for a particular purpose. 

For example, this might include, but is not limited to, the state providing 
technical assistance to districts implementing a new teacher evaluation system. 

SCI 3. Does your state education agency currently work with any intermediaries 
to support the implementation of statewide education reform priorities in any of 
the following areas? For each reform area selected, please briefly describe the 
role assigned to intermediaries in that area. a. Implementing college and career 
ready standards and assessments; b. Using data to improve instruction; c. 
Recruiting, developing, rewarding, and retaining effective teachers and school 
leaders; d. Turning around your state's lowest-achieving schools; e. Providing 
supports for English language learners; f. Increasing state capacity in any of the 
areas just mentioned. 

Working with districts that have 
the highest achievement levels 
and seeking to replicate their 
practices statewide to 
implement the education reform 
plan 

SC2. Which of the following strategies is your state using to implement this 
reform plan? b. Working with districts that have the highest achievement levels 
and seeking to replicate their practices statewide 

Providing effective and efficient 
operations and processes for 
grant oversight and 
performance measure tracking 
and reporting 

SC6. To what extent does the state education agency play each of the following 
roles (a great extent, moderate extent, little extent, or not at all)? b. Creator and 
monitor of performance measures including assessing the extent to which 
districts and/or schools are attaining specific performance goals; c. Compliance 
monitor of reform priorities such as ensuring compliance with state or federal 
laws and regulations 

SCI 8. Since we last interviewed state representatives in spring 2012, has the 
state education agency implemented any of the following changes to support 
the implementation of state education reform priorities? These could include 
reform priorities funded by Race to the Top (if applicable), as well as statewide 
reform efforts funded in other ways. a. Created a new office(s) or department(s); 
b. Created new staff positions; c. Reorganized the structure of existing 
offices/departments; d. Changed laws, policies, or regulations (IF YES, 

SPECIFY) 

Involving teachers, teachers 
unions or associations, or 
school administrators in 
defining its education reform 
initiatives or priorities 

SC5. To what extent (great extent, moderate extent, little extent, or not at all) 
are each of the following individuals or groups involved in defining your state’s 
education reform initiatives or priorities? c. Teachers; d. State teachers union or 
association; i. School administrators 

Involving other stakeholders in 
defining its education reform 
initiatives or priorities 

SC5. To what extent (great extent, moderate extent, little extent, or not at all) 
are each of the following individuals or groups involved in defining your state’s 
education reform initiatives or priorities? a. Governor’s office; b. State 
legislature; e. Businesses; f. Civil rights leaders; g. Local school boards; h. State 
school board; j. Parent, student and community organizations such as parent 
teacher organizations, local education foundations, community based 
organizations, or advocacy groups; k. Institutes of higher education including 
teacher and principal preparation programs or schools of education 


Source: Interviews with state administrators in spring 2013 
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Table C.2. Interview questions addressing the standards and assessments 
(SA) topic area policies and practices 


Interview questions addressing the standards 
Policy or practice and assessments policy or practice 


RTT application criterion 

• Commitment to adopting a common set of high-quality standards, evidenced by participation in a 
consortium of states that is working jointly to develop and adopt a common set of K-12 standards 


Adopting the Common Core SA1 . Did your state adopt the Common Core State Standards in both 

State Standards (CCSS) in both English/language arts and math, math only, English/language arts only, or 

English/language arts and math neither subject? 

SA3. What proportion of your total English/language arts and math standards 
are state-specific (meaning they are not part of the Common Core State 
Standards)? Please report separate percentages for English/language arts and 
math. 

RTT application criterion 

• Commitment to improving the quality of assessments, evidenced by participation in a consortium of 
states that is working jointly to develop and implement common, high-quality assessments 

Participating in a consortium of SA7. Is your state participating in a consortium to develop assessments 

states to develop assessments aligned to [the Common Core State Standards/your current state standards]? 

aligned to CCSS 

Has specified the school year by 
which state plans to fully 
implement summative 
assessments being developed 
by the consortium 

RTT application criterion 

• Supporting the statewide transition to and implementation of K-12 standards that build toward college 
and career readiness by the time of high school graduation and high-quality assessments tied to these 
standards 


SA9. In what school year does your state plan to fully implement, as opposed 
to pilot test, the summative assessments in English/language arts and math 
that are being developed by the consortium (a) in which you are participating? 
Please report the year by which you plan to fully implement the summative 
assessments being developed by the consortium in both subjects. 


Supporting the implementation 
of CCSS by providing funds for 
additional staff or making new 
technology investments 


Supporting the implementation 
of the assessments associated 
with the CCSS by making new 
technology investments 


SA14. For which of the following purposes were these funds designated? e. 
Fund additional staff either externally contracted or hired internally 3 

SA15. Since spring 2012, has the state made investments in new technology 
to assist with implementation of [the Common Core State Standards/your 
current state standards]? 

SA17. Since spring 2012, has the state made investments in new technology 
to assist with implementation of the assessments associated with [the 
Common Core State Standards/your current state standards]? 


Requiring or supporting the use SA12. Since we last interviewed state representatives in spring 2012, has your 
of new instructional materials for state made any of the following types of changes to state policies and 

implementing the CCSS practices in response to the adoption of [the Common Core State 

Standards/your current state standards]? a. Required use of new curricula or 
textbooks 


SA14. For which of the following purposes were these funds designated? b. 
Curriculum/textbook purchase 3 

SA19. Since spring 2012, has the state provided any materials to help 
practitioners understand [the Common Core State Standards/your current 
state standards] and/or change instruction based on the standards? 
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Policy or practice 


Interview questions addressing the standards 
and assessments policy or practice 


Developing, supporting, or 
requiring new interim 
assessments associated with the 
CCSS 


SA12. Since we last interviewed state representatives in spring 2012, has your 
state made any of the following types of changes to state policies and 
practices in response to the adoption of [the Common Core State 
Standards/your current state standards]? b. Required use of new, state- 
approved interim assessments 


SA14. For which of the following purposes were these funds designated? c. 
Development or purchase of interim assessments or item banks; d. Purchase 
hardware or software needed to implement standards or assessments 3 


Changing high school exit SA12. Since we last interviewed state representatives in spring 2012, has your 

requirements or college entrance state made any of the following types of changes to state policies and 
requirements practices in response to the adoption of [the Common Core State 

Standards/your current state standards]? c. Changed credit or course 
requirements such as adopting use of competency-based credit for courses or 
subject knowledge; d. Changed the content of the state’s high school exit 
exam; e. Changed the performance standard on the high school exit exam; f. 
Changed the college entrance requirements for the state college/university 
system 


Changing policies about or 
providing funds for professional 
development, training, and 
technical assistance for teachers 
or school leaders to support 
implementation of CCSS 


SA12. Since we last interviewed state representatives in spring 2012, has your 
state made any of the following types of changes to state policies and 
practices in response to the adoption of [the Common Core State 
Standards/your current state standards]? g. Changed policies related to 
teachers and/or school leaders such as licensure, certification, or annual 
professional development requirements 


SA14. For which of the following purposes were these funds designated? a. 
Professional development 3 


SA21 . Since spring 2012, has the state provided any professional 
development, training, or technical assistance (either directly or through an 
intermediary) to districts and/or schools to support implementation of [the 
Common Core State Standards/your current state standards]? 


Supporting districts or schools in SA25. Since we last interviewed state representatives in spring 2012, has the 
implementing the CCSS with state provided any supports to districts and/or schools specifically designed to 

English language learners aid in the implementation of the state’s standards with English language 

learners? This might include, but is not limited to, funds, professional 
development, technical assistance, or materials to support use of standards 
with English language learners. 

Source: Interviews with state administrators in spring 2013. 

3 This item followed a screener question that asked “Since spring 2012, has the state provided any funds to districts 
and/or schools to support implementation of [the Common Core State Standards/your current state standards]?” 


C-5 




APPENDIX C 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table C.3. Interview questions addressing the data systems (DA) topic area 
policies and practices 


Policy or practice Interview questions addressing the 

data systems policy or practice 


RTT application criterion: 

• Fully implement a state longitudinal data system (SLDS) that includes all of the America COMPETES 
Act elements 


Having an SLDS 

DAI . Does your state currently have a statewide longitudinal data system? 

SLDS has student-level 
enrollment and program 
participation information 

DAI 6. Which of the following types of data specifically related to English language 
learner students are currently included in your state’s longitudinal data system(s)? 
a. Students currently identified as English language learners; b. Students formerly 
identified as English language learners; c. Students’ English language proficiency 
test scores; d. English language learners’ proficiency in their native (home) 
language; e. Student participation in bilingual education programs, defined as 
programs that focus on developing literacy in two languages, which include non- 
English speakers and may include English speakers, and which involve 
instruction in English and another language; f. Student participation in other 
educational programs specifically designed for English language learners; g. 
Student’s native language; h. Number of years the student has lived in U.S.; k. 

The number of years that students have received English language instructional 
services; 1. The length of time it took for an English language learner to exit 

English language learners status 

SLDS is linked to an early 
childhood data system 

DA2. We are now going to ask you about data systems other than your state’s K- 
12 system, and whether they are linked to your state's K-12 education 
longitudinal data system, which means that data stored in these other systems 
can be accessed through the K-12 system. For each of the following data 
systems please indicate if they are currently linked to the state’s K-12 longitudinal 
education data system, a. Early childhood 

SLDS is linked to a higher 
education data system 

DA2. We are now going to ask you about data systems other than your state’s K- 
12 system, and whether they are linked to your state's K-12 education 
longitudinal data system, which means that data stored in these other systems 
can be accessed through the K-12 system. For each of the following data 
systems please indicate if they are currently linked to the state’s K-12 longitudinal 
education data system, b. Higher education 


RTT application criterion: 

• Ensure that key stakeholders (for example, principals, teachers, parents, community members, 

administrators) have access to and use data from the SLDS to support decision makers in continuous 
improvement efforts in areas such as policy, instruction, operations, management, resource allocation, 
and overall effectiveness 


SLDS can be accessed by 
stakeholders 

DA3. Which of the following groups currently has access to data from the state 
longitudinal data system? a. State Department of Education staff; b. district staff; 
c. principals; d. teachers; e. colleges and universities for purposes other than 
research; f. external researchers 
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Policy or practice 

Interview questions addressing the 


data systems policy or practice 


DA10. For which of the following purposes are data in the state longitudinal data 
system currently used by state-level staff? a. To track overall school performance; 
b. to evaluate instructional programs; c. To inform professional development 
offerings such as identifying specific content or skills where teachers need 
assistance or support; d. To evaluate the success of professional development 
offerings for teachers or principals; e. To inform resource allocation, such as 
which schools and students receive which programs or which staff work with 
which students; f. To provide information to teachers about their students’ 
progress; g. To provide information to parents about the school or their children; 
h. To provide information to students about their own progress; i. To track 
students’ postsecondary enrollment and progress after high school graduation, 
such as credits earned in public colleges or universities in the state; j. To provide 
information to federal agencies (for example, EDFacts) 

RTT application criteria: 

• Increase the acquisition, adoption, and use of local instructional improvement systems; support 
participating LEAs and schools that use instructional improvement systems in providing effective 
professional development to teachers, principals, and administrators on how to use these systems 

• Make data from instructional improvement systems, together with SLDS data, available to researchers 
to evaluate the effectiveness of instructional materials, strategies, and approaches for educating 
different types of students 

Requiring districts to 
implement district data 
systems 

Providing funding, materials, 
training, technical assistance, 
or other supports to districts 
to encourage the use of data 
to improve instruction 

Source: Interviews with state administrators in spring 2013. 


DA1 1 . Does the state require districts to implement district state systems as 
defined above? (we will refer to local instructional improvement systems as district 
data systems, which are technologically based tools or strategies that provide 
educators with data to manage continuous instructional improvement efforts) 

DAI 2. Within the past year, has the state provided funding, materials, training, 
technical assistance, or other supports to districts to encourage the use of data to 
improve instruction? 


SLDS is used to inform and 
engage stakeholders and 
support decision makers in 
continuous improvement 
efforts 
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Table C.4. Interview questions addressing the teacher and principal 
certification and evaluation (TL) topic area policies and practices 


Interview questions addressing the teacher and principal certification 
Policy or practice and evaluation policy or practice 


RTT application criteria: 

• Legal, statutory, or regulatory provisions that allow alternative routes to certification for teachers and 
principals, particularly routes that allow for providers in addition to higher education institutions 

• Alternative routes to teacher and principal certification that are in use 

• A process for monitoring, evaluating, and identifying areas of teacher and principal shortage and for 
preparing teachers and principals to fill these areas of shortage 


Authorizing or expanding 
institutions qualified to operate 
alternative-route programs for 
teachers 


TL2. Currently, which of the following groups are authorized to operate 
alternative teacher certification programs in your state? a. Institutions of higher 
education; b. School districts; c. Educational service districts or cooperatives in 
your state; d. Nonprofit organizations such as Teach for America; e. 
Partnerships or collaborations between groups listed above 

TL5. Within the past year, did the state implement any of the following types of 
changes to state regulations or policies related to either traditional teacher 
preparation programs or alternative teacher certification programs in the state? 

a. Expanded the types of institutions qualified to operate teacher preparation 
programs to include providers operating independently of institutions of higher 
education 


Adopting policies to increase TL5. Within the past year, did the state implement any of the following types of 
the selectivity of alternative- changes to state regulations or policies related to either traditional teacher 

route programs for teachers preparation programs or alternative teacher certification programs in the state? 

b. Adopted policies or regulations designed to increase the selectivity of teacher 
preparation programs 


Adopting policies to increase 
the amount of mentoring for 
participants in teacher 
certification programs or the 
time such participants spend in 
school-based learning 
experiences 


TL5. Within the past year, did the state implement any of the following types of 
changes to state regulations or policies related to either traditional teacher 
preparation programs or alternative teacher certification programs in the state? 

c. Adopted policies or regulations designed to increase the amount of time 
students in teacher preparation programs spend in supervised, school-based 
learning experiences; d. Adopted policies or regulations to increase the amount 
of mentoring and/or coaching that students in teacher preparation programs 
receive 


Adopting policies allowing 
alternative-route programs for 
teachers to award the same 
type of certification as 
traditional preparation programs 


TL5. Within the past year, did the state implement any of the following types of 
changes to state regulations or policies related to either traditional teacher 
preparation programs or alternative teacher certification programs in the state? 
e. Adopted policies or regulations that allow alternative certification programs to 
award the same type of certification that traditional preparation programs award 


Groups are currently operating TL1 . Currently, does the state authorize or permit alternative routes to 

alternative-route programs for certification for teachers? 

teachers 

TL2. Currently, which of the following groups are authorized to operate 
alternative teacher certification programs in your state? 

TL3. Are they currently operating such programs in your state? a. Institutions of 
higher education; b. School districts; c. Educational service districts or 
cooperatives in your state; d. Nonprofit organizations such as Teach for 
America; e. Partnerships or collaborations between groups listed above 


Having a process to identify TL14. Does the state currently have a process to identify areas of teacher 
areas of teacher shortages shortages? 
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Interview questions addressing the teacher and principal certification 
Policy or practice and evaluation policy or practice 


Taking steps to address areas TL17. Within the past year, has the state taken any of the following steps to 
of teacher shortage address these areas of teacher shortage? a. Enacted policies allowing 

traditional teacher preparation programs to expand the number of students 
served; b. Enacted policies allowing alternative teacher preparation programs to 
expand the number of students served; c. Provided incentives for teachers to 
become certified in areas of shortage (SPECIFY); d. Notified teacher 
preparation programs about the areas of shortage; e. Encouraged teacher 
preparation programs to train more teachers in the areas of shortage; f. 
Developed or began developing new alternative teacher certification programs 
focused on the areas of shortage; g. Increased the amount of induction support 
for novice teachers with the goal of reducing teacher attrition in areas of 
shortage 


Authorizing or expanding 
institutions qualified to operate 
alternative-route programs for 
principals 


TL19. Currently, which of the following groups are authorized to operate 
alternative principal certification programs in your state? a. Institutions of higher 
education; b. School districts; c. Educational service districts or cooperatives in 
your state; d. Nonprofit organizations such as Teach for America; e. 
Partnerships or collaborations between groups listed above 

TL22. Within the past year, did the state implement any of the following types of 
changes to state regulations or policies related to either traditional principal 
preparation programs or alternative principal certification programs in the state? 

a. Expanded the types of institutions qualified to operate principal preparation 
programs to include providers operating independently of institutions of higher 
education 


Adopting policies to increase 
the selectivity of alternative- 
route programs for principals 


Adopting policies to increase 
the amount of mentoring for 
participants in principal 
certification programs or the 
time such participants spend in 
school-based learning 
experiences 

Adopting policies allowing 
alternative-route programs for 
principals to award the same 
type of certification as 
traditional preparation programs 


TL22. Within the past year, did the state implement any of the following types of 
changes to state regulations or policies related to either traditional principal 
preparation programs or alternative principal certification programs in the state? 

b. Adopted policies or regulations designed to increase the selectivity of 
principal preparation programs 

TL22. Within the past year, did the state implement any of the following types of 
changes to state regulations or policies related to either traditional principal 
preparation programs or alternative principal certification programs in the state? 

c. Adopted policies or regulations designed to increase the amount of time 
students in principal preparation programs spend in supervised, school-based 
learning experiences 


TL22. Within the past year, did the state implement any of the following types of 
changes to state regulations or policies related to either traditional principal 
preparation programs or alternative principal certification programs in the state? 
e. Adopted policies or regulations that allow alternative certification programs to 
award the same type of certification that traditional preparation programs award 


Groups are currently operating 
alternative-route programs for 
principals 


Having a process to identify 
areas of principal shortages 


TL18. Currently, does the state authorize or permit alternative routes to 
certification for principals? 

TL19. Currently, which of the following groups are authorized to operate 
alternative principal certification programs in your state? 

TL20. Are they currently operating such programs in your state? a. Institutions 
of higher education; b. School districts; c. Educational service districts or 
cooperatives in your state; d. Nonprofit organizations such as Teach for 
America; e. Partnerships or collaborations between groups listed above 

TL31 . Does the state currently have a process to identify areas of principal 
shortages? 
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Interview questions addressing the teacher and principal certification 
Policy or practice and evaluation policy or practice 


Taking steps to address areas TL34. Within the past year, has the state taken any of the following steps to 

of principal shortage address these areas of principal shortage? a. Enacted policies allowing 

traditional principal preparation programs to expand the number of students 
served; b. Enacted policies allowing alternative principal preparation programs 
to expand the number of students served; c. Provided incentives for principals 
to become certified in areas of shortage (SPECIFY); d. Notified principal 
preparation programs about the areas of shortage; e. Encouraged principal 
preparation programs to train more principals in the areas of shortage; f. 
Developed or began developing new alternative principal certification programs 
focused on the areas of shortage; g. Increased the amount of induction support 
for novice principals with the goal of reducing principal attrition in areas of 
shortage 

RTT application criteria: 

• Design and implement rigorous, transparent, and fair evaluation systems for teachers and principals 
that differentiate effectiveness using multiple rating categories that take into account data on student 
growth as a significant factor 

• Conduct annual evaluations of teachers and principals that include timely and constructive feedback 

• Use evaluations, at minimum, to inform decisions regarding: developing teachers and principals, 
including by providing relevant coaching, induction support, or professional development; 
compensating, promoting, and retaining teachers and principals, including by providing opportunities 
for highly effective teachers and principals to obtain additional compensation and be given additional 
responsibilities; removing ineffective tenured and untenured teachers and principals after they have 
had ample opportunities to improve and ensuring that such decisions are made using rigorous 
standards and streamlined, transparent, and fair procedures 


Requiring districts to use 
student growth to evaluate 
teachers and specifying the 
extent to which student 
achievement growth must factor 
into teacher evaluations 


TL38. Currently, do state regulations specify the extent to which evidence of 
student growth must factor into the overall teacher evaluation? For example, 
regulations may call for it to be a “significant” factor in evaluations or call for 
student growth to have a specific weight such as 20 percent. If this is specified 
in state regulations, please describe those specific regulations. 

[Note: TL36 (shown in the next row) was also used to address the policy in this 
row. Specifically, this policy was coded as 0 if either the response to TL38 was 
“no” or the response to TL36 was “no teachers.”] 


Requiring multiple performance TL36. Currently, does the state require that districts use student growth as one 

measures for teacher measure when evaluating all teachers, some teachers, or no teachers? 

evaluations r r r . , , 

TL42. Which of the following other measures of teacher performance are 

currently required by the state for teacher evaluations for teachers in tested 
grades and/or subjects? a. Classroom observations conducted by the principal; 
b. Classroom observations conducted by a school administrator other than the 
principal; c. Classroom observations conducted by someone other than a 
school administrator, such as a peer or mentor teacher; d. self-assessment; e. 
portfolios or other artifacts of teacher practice; f. Peer assessments other than 
classroom observations; g. Student work samples; h. Student surveys or other 
feedback; i. Parent surveys or other feedback 

TL43. Currently, which of the following measures of teacher performance are 
required by the state for evaluating teachers of non-tested subjects, non-tested 
grades, or other special circumstances such as team-teaching? a. Classroom 
observations conducted by the principal; b. Classroom observations conducted 
by a school administrator other than the principal; c. Classroom observations 
conducted by someone other than a school administrator, such as a peer or 
mentor teacher; d. self-assessment; e. portfolios or other artifacts of teacher 
practice; f. Peer assessments other than classroom observations; g. Student 
work samples; h. Student surveys or other feedback; i. Parent surveys or other 
feedback 


C-10 





APPENDIX C 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Interview questions addressing the teacher and principal certification 
Policy or practice and evaluation policy or practice 


Specifying a required minimum TL44. Do state regulations specify a required minimum number of rating levels 
number of rating categories to to be used when evaluating overall teacher performance? If so, what is the 
be used when evaluating minimum number of rating categories that is required? 

teachers 


Conducting annual evaluations 
of teachers 


Requiring teacher evaluations 
to inform decisions about 
professional development and 
support for individual teachers 


Requiring teacher evaluations 
to inform decisions about 
compensation 


Requiring teacher evaluations 
to inform decisions about career 
advancement 


Requiring teacher evaluations 
to inform decisions about 
dismissal of teachers 


Requiring districts to use 
student growth to evaluate 
principals and specifying the 
extent to which student 
achievement growth must factor 
into principal evaluations 


Requiring multiple performance 
measures for principal 
evaluations 


Specifying a required minimum 
number of rating categories to 
be used when evaluating 
principals 


TL40. Currently, how often does the state require that teachers be evaluated 
during their probationary period? For example, every six months, annually, 
every other year, or at some other interval? 

TL41. Currently, how often does the state require that teachers be evaluated in 
their non-probationary period? For example, every six months, annually, every 
other year, or at some other interval? 

TL60. Currently, does the state prohibit, permit, or require teacher evaluation 
results to be used for any of the following purposes? Please select just one 
answer for each potential use. If your state both permits and requires that 
evaluation results be used for a particular purpose, please report that purpose 
as required, a. To guide decisions about professional development and support 
for individual teachers 

TL60. Currently, does the state prohibit, permit, or require teacher evaluation 
results to be used for any of the following purposes? Please select just one 
answer for each potential use. If your state both permits and requires that 
evaluation results be used for a particular purpose, please report that purpose 
as required, c. To guide decisions about annual teacher salary increases; d. To 
guide decisions about teacher bonuses or performance-based compensation 
other than salary 

TL60. Currently, does the state prohibit, permit, or require teacher evaluation 
results to be used for any of the following purposes? Please select just one 
answer for each potential use. If your state both permits and requires that 
evaluation results be used fora particular purpose, please report that purpose 
as required, e. To guide decisions about career advancement opportunities 

TL60. Currently, does the state prohibit, permit, or require teacher evaluation 
results to be used for any of the following purposes? Please select just one 
answer for each potential use. If your state both permits and requires that 
evaluation results be used for a particular purpose, please report that purpose 
as required, b. To guide decisions about dismissal of teachers 

TL75. Do state regulations specify the extent to which evidence of student 
growth must factor into the principal evaluation? For example, regulations may 
call for student growth to be a “significant” factor in evaluations or call for 
student growth to have a specific weight (such as 20 percent) in the principal 
evaluation. If this is specified in state regulations, please describe those specific 
regulations. 

[Note: TL74 (shown in the next row) was also used to address the policy in this 
row. Specifically, this policy was coded as 0 if either the response to TL75 was 
“no” or the response to TL74 was “no principals.”] 

TL74. Currently, does the state require that all districts use student growth as 
one measure when evaluating all principals, some principals, or no principals? 

TL77. Other than the student growth measure we just discussed, which of the 
following other measures of principal performance are currently required by the 
state? a. Self-assessment; b. District administrator input; c. Staff surveys or 
other feedback; d. Student surveys or other feedback 

TL78. Do state regulations specify a required minimum number of rating levels 
to be used when evaluating principal performance? If so, what is the minimum 
number of rating categories that is required? 
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Interview questions addressing the teacher and principal certification 
Policy or practice and evaluation policy or practice 


Requiring principal evaluations 
to inform decisions about 
professional development and 
support for individual principals 


TL89. Currently, does the state prohibit, permit, or require principal evaluation 
results to be used for any of the following purposes? Please select just one 
answer for each potential use. If your state both permits and requires that 
evaluation results be used for a particular purpose, please report that purpose 
as required, a. To guide decisions about professional development and support 
for principals 


Requiring principal evaluations TL89. Currently, does the state prohibit, permit, or require principal evaluation 

to inform decisions about results to be used for any of the following purposes? Please select just one 

compensation answer for each potential use. If your state both permits and requires that 

evaluation results be used for a particular purpose, please report that purpose 
as required, c. To guide decisions about principal salary increases; d. To guide 
decisions about principal bonuses or performance-based compensation other 
than salary 


Requiring principal evaluations 
to inform decisions about career 
advancement 


TL89. Currently, does the state prohibit, permit, or require principal evaluation 
results to be used for any of the following purposes? Please select just one 
answer for each potential use. If your state both permits and requires that 
evaluation results be used for a particular purpose, please report that purpose 
as required, e. To guide decisions about career advancement opportunities 


Requiring principal evaluations TL89. Currently, does the state prohibit, permit, or require principal evaluation 

to inform decisions about results to be used for any of the following purposes? Please select just one 

dismissal of principals answer for each potential use. If your state both permits and requires that 

evaluation results be used fora particular purpose, please report that purpose 
as required, b. To guide decisions about dismissal of principals. 

RTT application criterion: 

• Ensure the equitable distribution of teachers and principals by developing a plan to ensure that 
students in high-poverty or high-minority schools have equitable access to highly effective teachers 
and principals and are not served by ineffective teachers and principals at higher rates than other 
students 


Requiring districts or schools to 
use strategies to promote a 
more equitable distribution of 
effective teachers 


TL68. Currently, does your state require that districts and/or schools use any of 
the following strategies aimed at promoting a more equitable distribution of 
effective teachers between high-poverty or high-minority schools and low- 
poverty or low-minority schools? a. Offer more compensation for staff who 
teach at high-poverty or high-minority schools; b. Offer performance-based 
compensation for effective teachers who teach at high-poverty or high-minority 
schools; c. Hire staff in high-poverty or high-minority schools and districts 
earlier; d. Increase external recruitment activities such as open houses, job 
fairs, presentations, or advertisements; e. Improve teaching and learning 
environments at high-poverty or high-minority schools such as improved 
opportunities for collaboration, improved school leadership, or improved facility 
quality; f. Offer more professional development, mentoring, or induction for 
teachers working at high-poverty or high-minority schools 


Conducting analyses of teacher 
effectiveness based on student 
achievement growth to 
determine whether there has 
been a shift in the distribution of 
effective teachers 


TL69. Within the past year, has your state conducted analyses to determine 
whether there has been a shift in the distribution of effective teachers between 
high-poverty or high-minority schools and low-poverty or low-minority schools? 

TL70. Which of the following factors were examined in these analyses? a. 
Teacher effectiveness based on student growth such as average teacher 
effectiveness ratings in schools, proportion of teachers rated as highly effective 
in schools, or the distribution of teachers by rating categories in schools 
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Interview questions addressing the teacher and principal certification 
Policy or practice and evaluation policy or practice 


Requiring districts or schools to 
use strategies to promote a 
more equitable distribution of 
effective principals 


TL94. Currently, does your state require that districts and/or schools use any of 
the following strategies aimed at promoting a more equitable distribution of 
effective principals between high-poverty or high-minority schools and low- 
poverty or low-minority schools? a. Offer more compensation for principals who 
work at high-poverty or high-minority schools; b. Offer performance-based 
compensation for effective principals of high-poverty or high-minority schools; c. 
Hire staff in high-poverty or high-minority schools and districts earlier; d. 
Increase external recruitment activities such as open houses, job fairs, 
presentations, or advertisements; e. Improve teaching and learning 
environments at high-poverty or high-minority schools such as improved 
opportunities for collaboration, improved school leadership, or improved facility 
quality; f. Offer more professional development for principals working at high- 
poverty or high-minority schools, such as training, coaching, or principal 
academies 


Conducting analyses of 
principal effectiveness based on 
student achievement growth to 
determine whether there has 
been a shift in the distribution of 
effective principals 


TL95. Within the past year, has your state conducted analyses to determine 
whether there has been a shift in distribution of effective principals between 
high-poverty or high-minority schools and low-poverty or low-minority schools? 

TL96. Which of the following factors were examined in these analyses? b. 
Principal effectiveness ratings based on student growth 


RTT application criteria: 

• Link student achievement and student growth data to the students’ teachers and principals and in-state 
teacher and principal preparation programs and publicly report the data for each credentialing program 
in the state 

• Expand preparation and credentialing options and programs that are successful at producing effective 
teachers and principals 


Using teacher effectiveness 
ratings (based in part on 
student achievement growth) to 
assess the effectiveness of 
teacher certification programs 


Publicly reporting results from 
its evaluations of teacher 
certification program 
effectiveness 


Using results from its 
evaluations of teacher 
certification programs to provide 
additional funds for, expand, or 
promote certification programs 
that were shown to be effective 


Using principal effectiveness 
ratings (based in part on 
student achievement growth) to 
assess the effectiveness of 
principal certification programs 


TL1 1 . Currently, which of the following types of information does the state use 
to assess the effectiveness of teacher certification programs? Please indicate if 
each type of information is used for assessing effectiveness of traditional 
certification programs only, alternative certification programs only, both 
traditional and alternative programs, or neither, d. The effectiveness ratings 
(based in part on student growth) of credentialed teachers from each program 

TL12. Within the past year, did the state use the results from its evaluations of 
teacher certification programs in any of the following ways? For each potential 
use, please indicate if it was used for traditional certification programs only, 
alternative certification programs only, both traditional and alternative programs, 
or neither, f. Publicly reported results for each program 

TL12. Within the past year, did the state use the results from its evaluations of 
teacher certification programs in any of the following ways? For each potential 
use, please indicate if it was used for traditional certification programs only, 
alternative certification programs only, both traditional and alternative programs, 
or neither, b. Provided additional state funding to certification programs that 
were shown to be effective; c. Expanded and/or promoted certification 
programs that were shown to be effective 

TL28. Currently, which of the following types of information does the state use 
to assess the effectiveness of principal certification programs? Please indicate 
if each type of information is used for assessing effectiveness of traditional 
certification programs only, alternative certification programs only, both 
traditional and alternative programs, or neither, d. The effectiveness ratings 
(based in part on student growth) of credentialed principals from each program 
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Interview questions addressing the teacher and principal certification 
Policy or practice and evaluation policy or practice 


Publicly reporting results from 
its evaluations of principal 
certification program 
effectiveness 


TL29. Within the past year, did the state use the results from its evaluations of 
principal certification programs in any of the following ways? For each potential 
use, please indicate if it was used for traditional certification programs only, 
alternative certification programs only, both traditional and alternative programs, 
or neither, f. Publicly reported results for each program 


Using results from its 
evaluations of principal 
certification programs to provide 
additional funds, expand, or 
promote certification programs 
that were shown to be effective 


TL29. Within the past year, did the state use the results from its evaluations of 
principal certification programs in any of the following ways? For each potential 
use, please indicate if it was used for traditional certification programs only, 
alternative certification programs only, both traditional and alternative programs, 
or neither, b. Provided additional state funding to certification programs that 
were shown to be effective; c. Expanded and/or promoted certification 
programs that were shown to be effective 


Source: Interviews with state administrators in spring 2013. 


C-14 




APPENDIX C 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Table C.5. Interview questions addressing the school turnaround (TA) topic 
area policies and practices 


Policy or practice Interview questions addressing the school turnaround policy or practice 


RTT application criterion: 

• Intervene directly in the lowest-achieving schools and LEAs that are in improvement or corrective 
action status 


Having the authority to take TA43. Does your state currently have the authority to take over failing schools? 
over failing schools 

RTT application criterion: 

• Support LEAs whose lowest-achieving schools are implementing one of the four school intervention 
models: turnaround model, restart model, school closure, or transformation model 


Providing training to the 
lowest-achieving schools or 
LEAs on analyzing student 
assessment data to improve 
instruction 


TA29. Which of the following types of training and/or technical assistance has 
the state provided in the current school year to support the improvement efforts 
of the persistently lowest-achieving schools in your state and/or the districts in 
which these schools are located? Please report technical assistance provided 
directly by state Department of Education staff as well as technical assistance 
funded by the state but provided by someone other than state Department of 
Education staff, for example, an external consultant or staff from a regional 
office, a. Training on analyzing student assessment data to improve instruction 


Helping the lowest-achieving 
schools or LEAs align curricula 
to state standards 


TA29. Which of the following types of training and/or technical assistance has 
the state provided in the current school year to support the improvement efforts 
of the persistently lowest-achieving schools in your state and/or the districts in 
which these schools are located? e. Help aligning school curricula to state 
standards 


Providing training to the 
lowest-achieving schools or 
LEAs on identifying and 
implementing effective 
curricula, instructional 
strategies, or school 
intervention models, or 
developing and implementing 
a school improvement plan 

Providing training to the 
lowest-achieving schools or 
LEAs on identifying and 
implementing strategies to 
address the needs of ELLs 

Providing technical assistance 
to the lowest-achieving 
schools or LEAs on improving 
the quality of professional 
development 


TA29. Which of the following types of training and/or technical assistance has 
the state provided in the current school year to support the improvement efforts 
of the persistently lowest-achieving schools in your state and/or the districts in 
which these schools are located? c. Training on developing and implementing a 
school improvement plan; d. Training on identifying and implementing effective 
curricula, instructional strategies, or school intervention models 


TA29. Which of the following types of training and/or technical assistance has 
the state provided in the current school year to support the improvement efforts 
of the persistently lowest-achieving schools in your state and/or the districts in 
which these schools are located? f. Training on identifying and implementing 
strategies to address the needs of English language learners 

TA29. Which of the following types of training and/or technical assistance has 
the state provided in the current school year to support the improvement efforts 
of the persistently lowest-achieving schools in your state and/or the districts in 
which these schools are located? g. Assistance on improving the quality of 
professional development 
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Policy or practice Interview questions addressing the school turnaround policy or practice 


Providing operational flexibility 
and support to lowest- 
achieving schools or LEAs 
with regard to staffing and 
budgeting 

TA29. Which of the following types of training and/or technical assistance has 
the state provided in the current school year to support the improvement efforts 
of the persistently lowest-achieving schools in your state and/or the districts in 
which these schools are located? h. Assistance on analyzing and revising 
budgets to use resources more effectively 

TA22. Do the state’s current teacher assignment laws or policies for persistently 
lowest-achieving schools include any of the following features? e. School 
discretion or authority to decide which staff to hire for the persistently lowest- 
achieving schools 

TA17. Does the state currently provide persistently lowest-achieving schools any 
flexibility with, or exemptions from, collective bargaining agreements or policies 
that guide staffing in your state’s schools? 

TA25. Thinking specifically about your state’s persistently lowest-achieving 
schools, does the state, the districts, or individual schools currently have primary 
responsibility for the following? a. Setting student discipline policies; b. 

Developing school budgets; c. Establishing the curriculum, including core texts; 
d. Setting student assessment policies except state mandated tests; e. Staff 
hiring, discipline, and dismissal; f. Determining the length of the school day; g. 
Determining the length of the school year; h. Setting requirements for 
professional development 

Implementing or providing 
technical assistance on 
strategies to recruit and retain 
effective teachers (such as 
financial incentives) 

TA29. Which of the following types of training and/or technical assistance has 
the state provided in the current school year to support the improvement efforts 
of the persistently lowest-achieving schools in your state and/or the districts in 
which these schools are located? i. Assistance on developing strategies to 
recruit and retain more effective teachers 

TA22. Do the state’s current teacher assignment laws or policies for persistently 
lowest-achieving schools include any of the following features? a. Financial 
incentives for teachers to begin or continue to work in the state’s persistently 
lowest-achieving schools; b. Other retention or recruitment efforts specifically 
targeted at the state’s persistently lowest-achieving schools (SPECIFY) 

Having teacher tenure rules 
that affect placement in or 
removal from the lowest- 
achieving schools 

TA22. Do the state’s current teacher assignment laws or policies for persistently 
lowest-achieving schools include any of the following features? f. Teacher tenure 
rules that affect placement in or removal from the persistently lowest-achieving 
schools 

Having state-level 
administrative structures 
intended to support school 
turnaround efforts 

TA13. Does the state currently have any organizational or administrative 
structures specifically intended to improve state capacity to support school 
turnaround efforts? 

TA32. Which of the following groups are responsible for monitoring the state’s 
persistently lowest-achieving schools? a. State department of education 

TA37. Does the state provide additional reporting or support for persistently 
lowest-achieving schools that miss improvement targets? 


Source: Interviews with state administrators in spring 2013 
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Table C.6. Interview questions addressing the charter schools (CH) topic 
area policies and practices 


Policy or practice Interview questions addressing the charter school policy or practice 


RTT application criterion: 

• Have a charter school law that does not prohibit or effectively inhibit increasing the number of high- 
performing charter schools in the state 

Having no restrictions on the CH3. Currently, does the state have any restrictions on the creation of new 

creation of new charter schools charter schools and/or charter enrollment? 

or charter enrollment 

RTT application criterion: 

• Have laws, statutes, regulations, or guidelines regarding how charter school authorizers approve, 
monitor, hold accountable, reauthorize, and close charter schools 


In considering applications for CH9. In considering applications for new charter schools, does your state give 

new charter schools, giving priority to schools that specifically propose to address the needs of and/or 

priority to schools that propose to explicitly target English language learners? 
address needs of or target ELLs 


Monitoring the academic CHI 3. On which of the following dimensions is charter school performance 

performance of charter schools currently monitored by the state or its agent(s)? a. Academic performance 


Monitoring the non-academic CHI 3. On which of the following dimensions is charter school performance 

performance of charter schools currently monitored by the state or its agent(s)? b. Enrollment; c. Proportion of 

different student populations served such as English language learners; d. 
Finances and budget; e. Facilities 

Source: Interviews with state administrators in spring 2013. 
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Table C.7. Interview questions addressing the ELL-focused policies and 
practices 


ELL-focused policy or 

practice Interview questions 


1 ELL-focused policy or 

1 practice 

Interview questions 

Prioritizing the adoption and 
implementation of supports to 
ELLs 

SC3. To what extent (great extent, moderate extent, little extent, or not at all) is 
each of the following educational reform initiatives a priority in your state for the 
current school year? 1. Providing supports to ELLs 

SC21 . Which of the following statements best characterizes how ELLs fit into 
your state’s current education reform priorities? 1- ELLs are an explicit, central 
priority of statewide reform efforts, 2- English language learners are an emerging 
priority of statewide reform efforts, 3- Statewide reform efforts are designed to 
address the needs of all students, including ELLs 

Providing targeted support or 
working with intermediaries to 
provide support to ELLs 

SC 8 .T 0 which of the following types of districts and/or schools has the state 
provided targeted support for implementation of statewide education reforms? By 
targeted supports, we mean supports that are provided to a specific group so 
schools and/or districts for a particular purpose, c. Districts and/or schools with 
high proportions of ELLs 

SCI 3. Does your state education agency currently work with any intermediaries 
to support the implementation of statewide education reform priorities in any of 
the following areas? For each reform area selected, please briefly describe the 
role assigned to intermediaries in that area. e. Providing supports for ELLs 

Implementing organizational 
or administrative changes to 
improve capacity to support 
ELLs 

SC22. Since we last interviewed state representatives in spring 2012, has the 
state education agency implemented any of the following organizational or 
administrative changes to improve its capacity to address the needs of ELLs? a. 
Increased number of state Department of Education staff with ELL expertise; b. 
Reorganized the structure of existing offices or department with ELL 
responsibility or increased the collaboration of staff across these offices or 
departments; c. Increased use of external consultants with ELL expertise; d. 
Increased ELL expertise within regional offices; e. Redefined specific policies to 
better meet the needs of ELLs 

Supporting districts or schools 
in implementing the Common 
Core State Standards with 

ELLs 

SA25. Since we last interviewed state representatives in spring 2012, has the 
state provided any supports to districts and/or schools specifically designed to 
aid in the implementation of the state’s standards with ELLs? This might include, 
but is not limited to, funds, professional development, technical assistance, or 
materials to support use of standards with ELLs 

State longitudinal data system 
has program participation 
information about ELLs 

DA16. Which of the following types of data specifically related to ELL students 
are currently included in your state’s longitudinal data system(s)? a. Students 
currently identified as ELLs; b. Students formerly identified as ELLs; c. Students’ 
English language proficiency test scores; d. ELLs’ proficiency in their native 
(home) language; e. Student participation in bilingual education programs, 
defined as programs that focus on developing literacy in two languages, which 
include non-English speakers and may include English speakers, and which 
involve instruction in English and another language; f. Student participation in 
other educational programs specifically designed for ELLs; g. Student’s native 
language; h. Number of years the student has lived in U.S.; k. The number of 
years that students have received English language instructional services; 1. The 
length of time it took for an ELL to exit ELL status 
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ELL-focused policy or 

practice Interview questions 


State longitudinal data system 
is used to inform and engage 
stakeholders and support 
decision makers in continuous 
improvement efforts for ELLs 

DAI 7. For which of the following purposes are data in the state longitudinal data 
system specifically related to ELLs currently used by state-level staff? a. To track 
the progress of current ELLs; b. To track the progress of former ELLs; c. To track 
the ELL status of students; d. To identify the needs of specific ELL populations 
such as refugees, migrant education students, or students who speak low- 
incidence languages; e. To identify the professional development needs of 
teachers of ELLs; f. To assess teacher effectiveness with ELLs; g. To provide 
information to the parents of ELLs about their children’s progress; h. To provide 
information to ELLs about their own progress; i. To make decisions about exiting 
students from ELL status 

Providing funding, materials, 
training, technical assistance, 
or other supports to districts to 
aid in the use of ELL-related 
data 

DAI 8. Within the past year, has the state provided funding, materials, training, 
technical assistance, or other supports to districts specifically designed to aid in 
the use of data related to ELLs? 

Teacher assignment laws or 
policies include financial 
incentives to recruit and retain 
teachers with ELL-expertise 

TA22. Do the state’s current teacher assignment laws or policies for persistently 
lowest-achieving schools include any of the following features? c. Financial 
incentives for staff with ELL expertise to begin or continue to work in persistently 
lowest-achieving schools; d. Other retention or recruitment efforts targeted 
toward increasing the number of staff with ELL expertise in the persistently 
lowest-achieving schools (SPECIFY) 

Providing training to the 
lowest-achieving schools or 
local education agencies on 
identifying and implementing 
strategies to address the need 
of ELLs 

TA29. Which of the following types of training and/or technical assistance has the 
state provided in the current school year to support the improvement efforts of 
the persistently lowest-achieving schools in your state and/or district in which 
these schools are located? f. Training on identifying and implementing strategies 
to address the needs of ELLs 

Having state-level staff or 
consultants to support 
turnaround schools and 
districts working with ELLs 

TA 14. Which of the following organizational or administrative structures are in 
place in your state for this purpose? f. State-level staff or consultants to provide 
support to turnaround schools and districts in working with ELLs 

In considering applications for 
new charter schools, giving 
priority to schools that propose 
to address needs of or target 
ELLs 

CH9. In considering applications for new charter schools, does your state give 
priority to schools that specifically propose to address the needs of and/or 
explicitly target ELLs? 

Monitoring charter school 
performance based on the 
student populations (such as 
ELLs) served 

CHI 3. On which of the following dimensions is charter school performance 
currently monitored by the state or its agent(s)? c. Proportion of different student 
populations served, such as ELLs 


Source: Interviews with state administrators in spring 2013 
ELL= English language learner. 
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In this appendix, we present figures showing changes between 2007-2008 and 2012-2013 
and between 2011-2012 and 2012-2013 in the extent to which states reported using RTT- 
promoted policies and practices. We focus on the same six topic areas and policies and practices 
that were the focus of Chapter IV and the same ELL-focused policies and practices that were the 
focus of Chapter V. In Section A, we present changes between 2007-2008 and 2012-2013, and 
in Section B, we present changes between 2011-2012 and 2012-2013. 

A. Changes between 2007-2008 and 2012-2013 

We examined changes between 2007-2008 and 2012-2013 for the three topic areas for 
which we had state reports for 2007-2008: (1) teacher and principal certification and evaluation, 
(2) school turnaround, and (3) charter schools. This analysis sheds light on the extent to which 
the three groups of states changed their use of policies and practices since baseline. However, 
this analysis has several caveats: (1) we only had baseline infonnation on policies and practices 
in three of the six areas, (2) for these three areas, we only had infonnation on a subset of policies 
and practices, (3) infonnation for 2007-2008 was collected retrospectively in spring 2012 and 
was not independently verified by the research team, so readers should exercise caution when 
interpreting data from 2007-2008. 

For these three areas, we found no differences between RTT and other states in the changes 
in the use of RTT-promoted policies and practices between 2007-2008 and 2012-2013: 

• Teacher and principal certification and evaluation. Among the 12 policies and practices 
for which we had data from 2007-2008, early and later RTT states had already reported 
using more of these policies and practices than non-RTT states at baseline. Each group 
increased their use of these policies and practices by 2012-2013, but the increases for early 
and later RTT states did not differ from the increase for non-RTT states by statistically 
significant amounts (Figure D.l). 

• School turnaround. Among the five policies and practices with baseline data, later RTT 
states had already reported using more of these policies and practices than non-RTT states at 
baseline. Each group of states increased their use of these policies and practices by 2012- 
2013, but the increases did not differ across RTT statuses by statistically significant amounts 
(Figure D.2). 

• Charter schools. Among the three policies and practices with baseline data, there were no 
statistically significant differences between RTT states and other states at baseline or in 
changes in states’ use of policies and practices over time (Figure D.3). 
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Figure D.l. Change between 2007-2008 and 2012-2013 in use of policies and 
practices aligned with RTT objectives on teacher and principal certification 
and evaluation 



Early RTT States Later RTT States _ Non-RTT States 
(N = 1 2) (N=7) (N=31 -32) 

Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: This figure shows change over time for each group of states in the use of policies and practices aligned 

with the teacher and principal certification and evaluation section of the RTT application criteria. The arrow 
for each group of states starts at the average number of reported policies and practices aligned with the 
RTT application criteria in 2007-2008 and ends at the average number of reported policies and practices 
aligned with the RTT application criteria in spring 2013. For example, on average, non-RTT states reported 
that they used 2.7 of the 12 policies and practices aligned with the overall teacher and principal evaluation 
and certification topic area in 2007-2008. This group of states, on average, reported using 4.4 of these 12 
policies and practices in spring 2012 and 4.5 of these 12 policies and practices in spring 2013. The average 
numbers for spring 2013 differ from those reported in Chapter IV because this figure focuses on just the 
subset of policies and practices (12 out of 39) that the interviews asked about for 2007-2008, 201 1-2012, 
and 2012-2013. There were no statistically significant differences between RTT and non-RTT states with 
respect to changes between 2007-2008 and 2012-2013 in the number of policies and practices used, at 
the 0.05 level using a two-tailed test. Data from 2007-2008 were collected retrospectively in spring 2012. 

All data provided by states were self-reported and not independently verified by the research team. For 
these reasons and potential concerns about recall accuracy, readers should exercise caution when 
interpreting data from 2007-2008. A range is provided for the sample sizes because nonresponse varied 
across years. 


D-3 



APPENDIX D 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure D.2. Change between 2007-2008 and 2012-2013 in use of policies and 
practices aligned with RTT objectives on school turnaround 
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Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: This figure shows change over time for each group of states in the use of policies and practices aligned 

with the school turnaround section of the RTT application. The arrow for each group of states starts at the 
average number of reported policies and practices aligned with the RTT application criteria in 2007-2008 
and ends at the average number of reported policies and practices aligned with the RTT application criteria 
in spring 2013. For example, on average, non-RTT states reported that they had used 0.8 of the 5 policies 
and practices aligned with the overall school turnaround topic area in 2007-2008. This group of states, on 
average, reported using 1.3 of these 5 policies and practices in spring 2012 and 1.3 of these policies and 
practices in spring 201 3. The average numbers for spring 201 3 differ from those reported in Chapter IV 
because this figure focuses on just the subset of policies and practices (5 out of 10) that the interviews 
asked about for 2007-2008, 2011-2012, and 2012-2013. A range is provided for the sample sizes 
because nonresponse varied across items. There were no statistically significant differences between RTT 
and non-RTT states with respect to changes between 2007-2008 and 2012-2013 in the number of policies 
and practices used, at the 0.05 level using a two-tailed test. Data from 2007-2008 were collected 
retrospectively in spring 2012. All data provided by states were self-reported and not independently verified 
by the research team. For these reasons and potential concerns about recall accuracy, readers should 
exercise caution when interpreting data from 2007-2008. A range is provided for the sample sizes because 
nonresponse varied across years. 
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Figure D.3. Change between 2007-2008 and 2012-2013 in use of policies and 
practices aligned with RTT objectives on charter schools 



2008 2012 2013 


Early RTT States Later RTT States Non-RTT States 

(N=1 2) (N=7) (N=29-32) 

Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: This figure shows change over time for each group of states in the number of policies and practices aligned 

with the charter school section of the RTT application. The arrow for each group of states starts at the 
average number of reported policies and practices aligned with the RTT application criteria in 2007-2008 
and ends at the average number of reported policies and practices aligned with the RTT application criteria 
in spring 2013. For example, on average, non-RTT states reported that they used 1 .3 of the 3 policies and 
practices aligned with the overall charter school topic area in 2007-2008. This group of states, on average, 
reported using 1.5 of these 3 policies and practices in spring 2012 and 1.6 of these 3 policies and practices 
in spring 2013. The average numbers for spring 2013 differ from those reported in Chapter IV because this 
figure focuses on just the subset of policies and practices (3 out of 4) for which we measured usage in 
2007-2008, 201 1-2012, and 2012-2013. A range is provided for the sample sizes because nonresponse 
varied across items. There were no statistically significant differences between RTT and non-RTT states 
with respect to changes between 2007-2008 and 2012-2013 in the number of policies and practices used, 
at the 0.05 level using a two-tailed test. Data from 2007-2008 were collected retrospectively in spring 2012. 
All data provided by states were self-reported and not independently verified by the research team. For 
these reasons and potential concerns about recall accuracy, readers should exercise caution when 
interpreting data from 2007-2008. A range is provided for the sample sizes because nonresponse varied 
across years. 
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B. Changes between 2011-2012 and 2012-2013 

We examined changes between 2011-2012 and 2012-2013 for all six topic areas and the 
ELL-focused policies and practices. Since some policies and practices may have taken time to 
implement, this analysis sheds light on the extent to which states increased their use of policies 
and practices during successive years of RTT implementation. On the other hand, if states 
quickly began using all policies that they intended to use, we might expect to see few changes 
across years. This analysis used data on all the policies and practices examined in Chapters IV 
and V. 

For all six areas, there were no significant differences between RTT and other states in the 
changes over time in use of RTT-promoted policies and practices (Figures D.4 through D.9). 
Similarly, the changes over time in use of EFF-focused policies and practices did not 
significantly differ between RTT and other states (Figure D.10). 

Figure D.4. Change between 2011-2012 and 2012-2013 in use of policies and 
practices aligned with RTT objectives on state capacity 
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Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: This figure shows change over time for each group of states in the number of policies and practices aligned 

with the state capacity section of the RTT application. The arrow for each group of states starts at the 
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average number of reported policies and practices aligned with the RTT application criteria in spring 2012 
and ends at the average number of reported policies and practices aligned with the RTT application criteria 
in spring 2013. For example, on average, early RTT states reported that they used 7.5 of the 10 policies 
and practices aligned with the overall state capacity topic area in spring 2012 and 7.1 of these policies and 
practices in spring 2013. There were no statistically significant differences between RTT and non-RTT 
states with respect to changes between 201 1-2012 and 2012-2013 in the number of policies and practices 
used, at the 0.05 level using a two-tailed test. A range is provided for the sample sizes because 
nonresponse varied across years. 

Figure D.5. Change between 2011-2012 and 2012-2013 in use of policies and 
practices aligned with RTT objectives on standards and assessments 



_ Early RTT States Later RTT States Non-RTT States 

(N=12) (N=7) (N=31 -32) 

Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: This figure shows change over time for each group of states in the number of policies and practices aligned 

with the standards and assessments section of the RTT application. The arrow for each group of states 
starts at the average number of reported policies and practices aligned with the RTT application criteria in 
spring 2012 and ends at the average number of reported policies and practices aligned with the RTT 
application criteria in spring 2013. For example, on average, non-RTT states reported using 4.0 of the 10 
policies and practices aligned with the overall standards and assessments topic area in spring 2012 and 4.6 
of these policies and practices in spring 2013. There were no statistically significant differences between 
RTT and non-RTT states with respect to changes between 201 1-2012 and 2012-2013 in the number of 
policies and practices used, at the 0.05 level using a two-tailed test. A range is provided for the sample 
sizes because nonresponse varied across years. 
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Figure D.6. Change between 2011-2012 and 2012-2013 in use of policies and 
practices aligned with RTT objectives on data systems 



Early RTT States _ Later RTT States _ Non-RTT States 
(N=12) (N=7) (N=31-32) 

Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: This figure shows change over time for each group of states in the number of policies and practices aligned 

with the data systems section of the RTT application. The arrow for each group of states starts at the 
average number of reported policies and practices aligned with the RTT application criteria in spring 2012 
and ends at the average number of reported policies and practices aligned with the RTT application criteria 
in spring 2013. For example, on average, non-RTT states reported that they used 5.1 of the 8 policies and 
practices aligned with the overall data systems topic area in spring 2012 and 5.8 of these 8 policies and 
practices in spring 2013. There were no statistically significant differences between RTT and non-RTT 
states with respect to changes between 201 1-2012 and 2012-2013 in the number of policies and practices 
used, at the 0.05 level using a two-tailed test. A range is provided for the sample sizes because 
nonresponse varied across years. 
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Figure D.7. Change between 2011-2012 and 2012-2013 in use of policies and 
practices aligned with RTT objectives on teacher and principal certification 
and evaluation 



Early RTT States Later RTT States Non-RTT States 
(N=1 2) ’ (N-7) (N=31 -32) 

Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: This figure shows change over time for each group of states in the use of policies and practices aligned 

with the teacher and principal certification and evaluation section of the RTT application criteria. The arrow 
for each group of states starts at the average number of reported policies and practices aligned with the 
RTT application criteria in spring 2012 and ends at the average number of reported policies and practices 
aligned with the RTT application criteria in spring 2013. For example, on average, non-RTT states reported 
that they had used 7.4 of the 39 policies and practices aligned with the overall teacher and principal 
evaluation and certification topic area in spring 2012 and 7.8 of these policies and practices in spring 2013. 
There were no statistically significant differences between RTT and non-RTT states with respect to 
changes between 201 1-2012 and 2012-2013 in the number of policies and practices used, at the 0.05 
level using a two-tailed test. A range is provided for the sample sizes because nonresponse varied across 
years. 
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Figure D.8. Change between 2011-2012 and 2012-2013 in use of policies and 
practices aligned with RTT objectives on school turnaround 
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Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: This figure shows change over time for each group of states in the number of policies and practices aligned 

with the school turnaround section of the RTT application criteria. The arrow for each group of states starts 
at the average number of reported policies and practices aligned with the RTT application criteria in spring 
2012 and ends at the average number of reported policies and practices aligned with the RTT application 
criteria in spring 2013. For example, on average, non-RTT states reported using 5.8 of the 10 policies and 
practices aligned with the overall school turnaround topic area in spring 2012 and 5.3 of these policies and 
practices in spring 2013. There were no statistically significant differences between RTT and non-RTT 
states with respect to changes between 201 1-2012 and 2012-2013 in the number of policies and practices 
used, at the 0.05 level using a two-tailed test. A range is provided for the sample sizes because 
nonresponse varied across years. 
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Figure D.9. Change between 2011-2012 and 2012-2013 in use of policies and 
practices aligned with RTT objectives on charter schools 


Overall Charter Schools Topic 
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(N=1 2) (N=7) (N=31 -32) 

Source: Interviews with state administrators in spring 2012 and spring 2013. 

Note: This figure shows change over time for each group of states in the number of policies and practices aligned 

with the charter school section of the RTT application criteria. The arrow for each group of states starts at 
the average number of reported policies and practices aligned with the RTT application criteria in spring 
2012 and ends at the average number of reported policies and practices aligned with the RTT application 
criteria in spring 2013. For example, on average, non-RTT states reported that they used 1.5 of the 4 
policies and practices aligned with the overall charter school topic area in spring 2012 and 1.6 of these 
policies and practices in spring 2013. There were no statistically significant differences between RTT and 
non-RTT states with respect to changes between 201 1-2012 and 2012-2013 in the number of policies and 
practices used, at the 0.05 level using a two-tailed test. A range is provided for the sample sizes because 
nonresponse varied across years. 
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Figure D.10. Change between 2011-2012 and 2012-2013 in use of ELL- 
focused policies and practices aligned with RTT objectives 



2012 2013 


Early RTT States _ Later RTT States _ Non-RTT States 
(N=1 2) (N=7) (N=31 -32) 

Source: Interviews with state administrators in spring 2012 and 2013. 

Note: This figure shows change over time for each group of states in the use of ELL-focused policies and 

practices aligned with the RTT application criteria. The arrow for each group of states starts at the average 
number of reported policies and practices aligned with the RTT application criteria in spring 2012 and ends 
at the average number of reported policies and practices aligned with the RTT application criteria in spring 
2013. For example, on average, non-RTT states reported using 5.6 of the 12 ELL-focused policies and 
practices aligned with the RTT application criteria in spring 2012 and 5.4 of these policies and practices in 
spring 2013. There were no statistically significant differences between RTT and non-RTT states with 
respect to changes between 201 1-2012 and 2012-2013 in the number of ELL-focused policies and 
practices used, at the 0.05 level using a two-tailed test. A range is provided for the sample sizes because 
nonresponse varied across years. 
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As described in Chapter I, RTT was designed to promote systemic change in the U.S. 
educational system, with the goal of improving student achievement. The general theory of 
action underlying RTT is that policy changes at the state level (which we examined in Chapters 
IV and V) represent the first step toward changing the education system. Changes would be 
expected to occur at the state level before occurring at lower levels — such as districts, schools, 
and classrooms. One mechanism by which states could implement these policy changes is by 
using RTT funds to increase spending; for example, on curriculum development and staff 
training. Student achievement could then improve due to the changes made at all levels of the 
education system. 

Recognizing that RTT represents a significant investment in education refonn, we first 
attempted to estimate the effect of RTT on student outcomes using a comparative short 
interrupted time series design. This design compared how actual post-RTT outcomes in RTT and 
other states differed from projected outcomes based on pre-RTT patterns (Somers et al. 2013). 
However, this approach did not produce credible results (see Chapter II for more discussion). 
Therefore, to avoid misinterpretation of the results from this approach, we proceeded with a 
purely descriptive analysis of the relationship between receipt of an RTT grant and student 
achievement. 

In this appendix, we present our descriptive analysis of the relationship between receipt of 
an RTT grant and student achievement. This analysis displays student- and state-level outcomes 
before and after the award of RTT grants. Specifically, we plotted outcomes over time for three 
groups of states (early RTT states, later RTT states, and non-RTT states). For each group of 
states, we plotted the average outcome across all states in that group for each year. 

The primary outcome we examined in this analysis was state-level mean NAEP scores 
because NAEP scores are consistent across states and over the time frame covered by our study, 
whereas states may have changed their own achievement measures during the years following 
RTT awards. As an interim outcome, we examined state per-pupil spending from the Common 
Core of Data (CCD). We also examined several outcomes focused on English language learner 
(ELL) students: ELL subgroup NAEP scores and ELL achievement gaps in NAEP scores. 

In Section A, we illustrate how to interpret the figures, using the outcomes of 4th grade math 
NAEP scores and per-pupil spending. In Section B, we present figures for 4th grade reading, 8th 
grade math, and 8th grade reading NAEP scores. In Section C, we present figures for NAEP 
scores for particular student subgroups. These subgroups are of interest because one of the goals 
of the RTT program was to support states and districts in closing achievement gaps between 
high-need students (for example, those eligible for free or reduced-price lunch) and those who 
are less disadvantaged (U.S. Department of Education n.d.). Specifically, for each of the NAEP 
outcomes (4th grade math, 4th grade reading, 8th grade math, and 8th grade reading), we present 
a series of figures showing the pattern of the outcome over time for the following subgroups: 

• Males 

• Females 

• White students 

• Black students 
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• Hispanic students 

• Students eligible for free or reduced-price lunch (FRPL) 

In Section D, we present figures for outcomes focused on ELL students. First, for each of the 
NAEP outcomes, we present a figure showing the pattern of the outcome over time for the 
subgroup of ELL students. Then, for each of the NAEP outcomes, we present a figure showing 
the NAEP score achievement gap between ELLs and other students over time. 

A. The relationship between RTT and student outcomes was not clear 

The relationship between RTT and student outcomes was not clear because different, 
equally valid, interpretations of the pattern of outcomes in the pre-RTT period yielded 
conflicting findings. 

As an example, Figure E. 1 shows how findings for 4th grade math NAEP scores for early 
RTT versus non-RTT states could be interpreted in three ways: 

1 . One interpretation of the pattern of outcomes in the pre-RTT period in Figure E. 1 would 
suggest RTT had a positive impact. In particular, it is possible that early RTT states were 
catching up to non-RTT states during the 2003-2009 pre-RTT period, but had ceased 
catching up by 2009. In this case, we would predict that the gap between early RTT and 
non-RTT states in 2009 would persist after 2009 if the RTT program had not been enacted. 
Instead, we see that the early RTT states continued to catch up after 2009, resulting in a 
smaller gap in 2015. This reduction in the gap would be consistent with RTT having a 
positive impact. 

2. An alternative interpretation would suggest that RTT had no effect. Specifically, it is 
possible that early RTT states were steadily catching up to non-RTT states each year from 
2003 to 2009, and that they would have continued to catch up at a similar pace even in the 
absence of RTT. In this case, even without the RTT program, we would expect early RTT 
states to continue catching up to the non-RTT states. This would be consistent with RTT 
having no effect. 

3. Yet another interpretation would suggest that RTT had a negative effect. Specifically, if 
early RTT states were catching up to non-RTT states at an accelerating pace in the pre-RTT 
period, and would have continued doing so at that faster pace in the absence of RTT, then 
one might conclude that RTT had a negative effect. 

All of these interpretations were plausible because it was difficult to determine the true 
pattern of outcomes in the pre-RTT period, due to the fact that NAEP data were only available 
for all states for four pre-RTT years. 

These interpretations illustrate the difficulty in providing credible estimates of the effect of 
RTT on 4th grade NAEP math scores. Trends in scores during the pre-RTT period could be 
interpreted as providing evidence of a positive effect of RTT, a negative effect of RTT, or no 
effect of RTT. 
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Figure E.l. 4th grade math NAEP scores for early, later, and non-RTT states, 
before and after the award of RTT grants 
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Source: NAEP scores. 

Notes: This figure shows 4th grade math NAEP scores from 2003-2015 (NAEP tests are administered every other 

year). A score of 214 is the cutoff for the “Basic” achievement level, which denotes partial mastery of 
required knowledge and skills. A score of 249 is the cutoff for the “Proficient” achievement level, which 
represents solid academic performance. The 2002-2003 school year is represented by 2003, the 2004- 
2005 school year by 2005, etc. The blue vertical line at 2010 represents the timing of RTT grants to early 
RTT states. The orange vertical line at 2012 represents the timing of RTT grants to later RTT states. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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In addition, the composition of students who took NAEP tests within each state varied from 
year to year. In other words, the 4th graders who took the NAEP math test in 2009 were different 
from the 4th graders who took the test in 20 1 1 . Differences in student characteristics could have 
contributed to differences in outcomes from year to year that may have little to do with the 
educational system or policies promoted by RTT. 

The conclusion for the other student outcomes we analyzed — NAEP 8th grade math scores, 
NAEP 4th and 8th grade reading scores, and the ELL-focused outcomes listed above — was the 
same. Namely, based on pre-RTT outcomes, it was not possible to detennine what would have 
happened in the post-RTT period in the absence of RTT. Therefore, the relationship between 
RTT and each of these student outcomes was not clear. 

The same uncertainty applies to estimates of the effect of RTT on the interim per-pupil 
spending outcome. Readers nevertheless may be interested to see the pattern of per-pupil 
spending before and after the award of RTT grants (Figure E.2). Before the award of RTT grants, 
early RTT states had higher per-pupil spending than non-RTT states. However, this gap in per- 
pupil spending did not appear to substantially widen after the award of RTT grants. We might 
expect to see a substantial widening of this gap in per-pupil spending if the RTT grants had 
supplemented state and local education funding. If the grants instead had supplanted state and 
local education funding (for example, if RTT funds essentially served to replace — but not add 
to — state and local funding that was lost during the severe recession that began in 2007), we 
would not expect to see a substantial widening of this gap in per-pupil spending between early 
RTT states and non-RTT states. 

We concluded that the relationship between RTT and student outcomes was not clear. 
Conclusions about the relationship depend on assumptions made about the pattern of outcomes 
in the pre-RTT period. However, with only four pre-RTT data points, it was difficult to 
detennine the true pattern of outcomes in that period. Furthermore, even if it were possible to 
discern this pattern, the results could not be used to detennine whether receipt of an RTT grant 
caused changes in outcomes because other changes happening at the same time as RTT reforms 
might also have affected outcomes. Nevertheless, we present the figures for the remaining 
outcomes and subgroups in the next three sections of this appendix, so readers can examine the 
patterns of outcomes on their own. 


E-5 



APPENDIX E 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.2. Per-pupil expenditures for early, later, and non-RTT states, before 
and after the award of RTT grants 

o 



x Early RTT (N=12) • Non-RTT (N=32) 


o 



Source: Common Core of Data (CCD). 

Notes: This figure depicts per-pupil current expenditures from 2004-2012, expressed in constant 2012 dollars. 

(Current expenditures, as defined by the CCD, include instruction, student support, instructional staff 
services, operation and maintenance, administration, transportation, and food services. 

RTT = Race to the Top 
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B. Figures for NAEP scores 

Figure E.3. 4th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants 
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Source: NAEP scores. 

Notes: A score of 208 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 238 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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Figure E.4. 8th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants 
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Source: NAEP scores. 

Notes: A score of 262 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 299 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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Figure E.5. 8th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants 
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Source: NAEP scores. 

Notes: A score of 243 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 281 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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C. Subgroup figures for NAEP scores 

Figure E.6. 4th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, male subgroup 
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Notes: A score of 214 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 249 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.7. 4th grade math NAEP scores for early, later and non-RTT states, before and after the award of 
RTT grants, female subgroup 
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Notes: A score of 214 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 249 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.8. 4th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, white subgroup 
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Source: NAEP scores. 

Notes: A score of 214 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 249 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.9. 4th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, black subgroup 
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achievement level, which denotes partial mastery of required knowledge and skills. A score of 249 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.10. 4th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, Hispanic subgroup 
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which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.11. 4th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, FRPL subgroup 
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Notes: A score of 214 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 249 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; FRPL = free or reduced-price lunch. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.12. 4th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, male subgroup 
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Source: NAEP scores. 

Notes: A score of 208 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 238 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.13. 4th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, female subgroup 
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Source: NAEP scores. 

Notes: A score of 208 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 238 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 


E-17 


APPENDIX E 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.14. 4th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, white subgroup 



X Early RTT (N=1 2) • Non-RTT (N=32) 


o 

^ - 
CM 


COO 
0 CO - 
il CM 

o 

o 

c 0 

CL ° 

Ml ^ 

LLj CM 

< 


-O CM 
(0 
<D 

cr 

©§ 


o 


: o 
: o> 


o 

co 


+ 


2003 


2005 


2007 


2009 


2011 


2013 


2015 


+ Later RTT (N=7) 


Non-RTT (N=32) 


Source: NAEP scores. 

Notes: A score of 208 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 238 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.15. 4th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, black subgroup 
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Source: NAEP scores. 

Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 208 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 238 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.16. 4th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, Hispanic subgroup 
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Source: NAEP scores. 

Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 208 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 238 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.17. 4th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, FRPL subgroup 
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Source: NAEP scores. 

Notes: A score of 208 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 238 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; FRPL = free or reduced-price lunch. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.18. 8th grade math NAEP scores for early, later, and non-RTT States, before and after the award of 
RTT grants, male subgroup 
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Source: NAEP scores. 

Notes: A score of 262 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 299 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.19. 8th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, female subgroup 
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Source: NAEP scores. 

Notes: A score of 262 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 299 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.20. 8th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, white subgroup 
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Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 262 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 299 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.21. 8th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, black subgroup 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.22. 8th grade math NAEP Scores for early, later, and non-RTT states, before and after the award of 
RTT grants, Hispanic subgroup 
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Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 262 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 299 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.23. 8th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, FRPL subgroup 
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Source: NAEP scores. 

Notes: A score of 262 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 299 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; FRPL = free or reduced-price lunch. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.24. 8th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, male subgroup 
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Source: NAEP scores. 

Notes: A score of 243 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 281 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.25. 8th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, female subgroup 
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Source: NAEP scores. 

Notes: A score of 243 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 281 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.26. 8th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, white subgroup 
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Source: NAEP scores. 

Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 243 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 281 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.27. 8th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, black subgroup 
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Source: NAEP scores. 

Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 243 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 281 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.28. 8th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, Hispanic subgroup 
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Source: NAEP scores. 

Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 243 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 281 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top. 


E-32 


APPENDIX E 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.29. 8th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, FRPL subgroup 
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Source: NAEP scores. 

Notes: A score of 243 is the cutoff for the “Basic” achievement level, which denotes partial mastery of required knowledge and skills. A score of 281 is the cutoff 

for the “Proficient” achievement level, which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; FRPL = free or reduced-price lunch. 
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RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


D. ELL-focused figures 

Figure E.30. 4th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, ELL subgroup 
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Source: NAEP scores. 

Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 214 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 249 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; ELL = English language learner. 


E-34 


APPENDIX E 


RACE TO THE TOP: IMPLEMENTATION AND RELATIONSHIP TO STUDENT OUTCOMES 


Figure E.31. 4th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, ELL subgroup 
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Source: NAEP scores. 

Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 208 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 238 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; ELL = English language learner. 
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Figure E.32. 8th grade math NAEP scores for early, later, and non-RTT states, before and after the award of 
RTT grants, ELL subgroup 
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Source: NAEP scores. 

Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 262 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 299 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; ELL = English language learner. 
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Figure E.33. 8th grade reading NAEP scores for early, later, and non-RTT states, before and after the award 
of RTT grants, ELL subgroup 
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Source: NAEP scores. 

Notes: A range is provided for the sample sizes because the number of states with missing data varied by year. A score of 243 is the cutoff for the “Basic” 

achievement level, which denotes partial mastery of required knowledge and skills. A score of 281 is the cutoff for the “Proficient” achievement level, 
which represents solid academic performance. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; ELL = English language learner. 
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Figure E.34. ELL achievement gap in 4th grade math NAEP scores for early, later, and non-RTT states, 
before and after the award of RTT grants 
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Source: NAEP scores. 

Notes: Each point on the figure represents the average difference between the scores of ELL students and non-ELL students for the group of states indicated. 

For each state, we computed the difference between the average score for ELL students and the average score for non-ELL students (ELL minus non- 
ELL). Negative numbers indicate that average scores for ELL students were lower than average scores for non-ELL students. We then took the average 
of these differences for each group of states (early, later, or non-RTT). A range is provided for the sample sizes because the number of states with 
missing data varied by year. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; ELL = English language learner. 
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Figure E.35. ELL achievement gap in 4th grade reading NAEP scores for early, later, and non-RTT states, 
before and after the award of RTT grants 
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Source: NAEP scores. 

Notes: Each point on the figure represents the average difference between the scores of ELL students and non-ELL students for the group of states indicated. 

For each state, we computed the difference between the average score for ELL students and the average score for non-ELL students (ELL minus non- 
ELL). Negative numbers indicate that average scores for ELL students were lower than average scores for non-ELL students. We then took the average 
of these differences for each group of states (early, later, or non-RTT). A range is provided for the sample sizes because the number of states with 
missing data varied by year. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; ELL = English language learner. 
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Figure E.36. ELL achievement gap in 8th grade math NAEP scores for early, later, and non-RTT states, 
before and after the award of RTT grants 
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Source: NAEP scores. 

Notes: Each point on the figure represents the average difference between the scores of ELL students and non-ELL students for the group of states indicated. 

For each state, we computed the difference between the average score for ELL students and the average score for non-ELL students (ELL minus non- 
ELL). Negative numbers indicate that average scores for ELL students were lower than average scores for non-ELL students. We then took the average 
of these differences for each group of states (early, later, or non-RTT). A range is provided for the sample sizes because the number of states with 
missing data varied by year. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; ELL = English language learner. 
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Figure E.37. ELL achievement gap in 8th grade reading NAEP scores for early, later, and non-RTT states, 
before and after the award of RTT grants 
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Source: NAEP scores. 

Notes: Each point on the figure represents the average difference between the scores of ELL students and non-ELL students for the group of states indicated. 

For each state, we computed the difference between the average score for ELL students and the average score for non-ELL students (ELL minus non- 
ELL). Negative numbers indicate that average scores for ELL students were lower than average scores for non-ELL students. We then took the average 
of these differences for each group of states (early, later, or non-RTT). A range is provided for the sample sizes because the number of states with 
missing data varied by year. 

NAEP = National Assessment of Educational Progress; RTT = Race to the Top; ELL = English language learner. 
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